Standby instance can't forward request with a message larger than 4MB

[ghost]

Describe the bug

With a standard Vault High Availability installation (with consul), the standby instance can't forward requests to transit secret engine if the message size is larger than 4MB. It switches automatically to the older/fallback redirection behavior (307 Temporary Redirect), which creates SSL verification issue

To Reproduce

Steps to reproduce the behavior:

1. Set 2 vault servers + consul like describe in official documentation : https://www.vaultproject.io/guides/operations/vault-ha-consul.html
2. Enable transit secret engine : https://www.vaultproject.io/docs/secrets/transit/index.html
3. We generate 2 files which will resend us data to provide to secret engine : 1 small (some bit), 1 large

# Generate test file
echo '{ "plaintext": "dGhlIHF1aWNrIGJyb3duIGZveA==" }' > small.json
echo '{ "plaintext": "' > large.json
cat /etc/hosts > large.tmp; for j in {1..6}; do for i in {1..5}; do cat large.tmp >> large2.tmp ; done; mv large2.tmp large.tmp; done;  cat large.tmp | base64 >> large.json ; rm large.tmp
echo '" }' >> large.json

# Check that large.json is > 4MB. In my case it's 7,6MB
ls -lh large.json

4. Test the small file on the active node and the standby, both should work and respond with a valid json response:

curl -L -X POST -H "X-Vault-Token:YOUR-TOKEN" --data @small.json https://vault-active.node.consul:8200/v1/transit/encrypt/testkey
# RECEIVED VALID JSON RESPONSE

curl -L -X POST -H "X-Vault-Token:YOUR-TOKEN" --data @small.json https://vault-standby.node.consul:8200/v1/transit/encrypt/testkey
#  RECEIVED VALID JSON RESPONSE

5. Test the large file. On the active node, you get the correct response, on the standby node, the connection terminates with 307 REDIRECT:

curl -L -X POST -H "X-Vault-Token:YOUR-TOKEN" --data @large.json https://vault-active.node.consul:8200/v1/transit/encrypt/testkey
# RECEIVED VALID JSON RESPONSE
curl -L -X POST -H "X-Vault-Token:YOUR-TOKEN" --data @large.json https://vault-standby.node.consul:8200/v1/transit/encrypt/testkey
# 307 Temporary Redirect + curl: (51) SSL: certificate subject name (*.service.consul) does not match target host name '192.168.X.X'

We don't have any message on the active server log, but we have this in the standby server log:

Jun 27 17:07:25 vault04-uk.decibelinsight.net vault[23663]: 2018-06-27T17:07:25.558+0100 [ERROR] core: error during forwarded RPC request: error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (9599688 vs. 4194304)"
Jun 27 17:07:25 vault04-uk.decibelinsight.net vault[23663]: 2018-06-27T17:07:25.558+0100 [ERROR] core: handleRequestForwarding: error forwarding request: error="error during forwarding RPC request"

Expected behavior

File > 4MB should have the same behavior. Switch on the legacy forwarding creates issue with HTTPS support

Environment:

• Vault Server Version: 0.10.2
• Vault CLI Version: Vault v0.10.2 ('3ee0802ed08cb7f4046c2151ec4671a076b76166')
• Server Operating System/Architecture: Ubuntu 16.04.4 LTS // kernel 4.4.0-121-generic

Vault server configuration file(s):

ui = true

storage "consul" {
  address = "192.168.X.X:8500"
  path = "vault"
}

listener "tcp" {
  address = "192.168.X.X:8200"
  #tls_disable=1
  tls_cert_file = "/etc/vault/certs/vault.crt"
  tls_key_file = "/etc/vault/certs/vault.key"
}

Additional context

It look like to be the default limitation of grpc protocol :
https://nanxiao.me/en/message-length-setting-in-grpc/
googleapis/google-cloud-node#1991

[jefferai]

It shouldn't be a problem

Maybe this actually has to be set on the server side too somehow?

[ghost]

Thanks for your quick feedback ! If you need more test/information, we are here :)