v1.4.0 service registration does not use addresses configured in the environment

[inkblot]

This is a regression

Bug Description
Vault 1.4.0 service registration with consul fails when service_address is not set in the service_registration block. The documentation for consul service registration indicates that it will use the HA redirect address as a fallback, which is unclear (there is nothing in the HA configuration parameters for the server that is described this way). However, I do have values configured for both HA addresses. The API address is configured using the VAULT_API_ADDR environment variable and the cluster address is configured using VAULT_CLUSTER_ADDR. This is a regression because in vault 1.3.0, service registration via the consul storage backend requires no additional configuration of the service address and correctly uses HA address information to register when it is specified in the environment.

To Reproduce
Steps to reproduce the behavior:

1. Set up a consul cluster
2. Run a vault servers configured to use consul for service registration, with HA configured via environment variables
3. See error in the server log at startup: [ERROR] service_registration.consul: error running service registration: redirect address must not be empty

Expected behavior
I expect the service registration to use whichever HA address is meant by "HA redirect address" (both have a working value) and successfully register with consul

Environment:

• Vault Server Version: 1.4.0
• Vault CLI Version: v1.4.0
• Server Operating System/Architecture: Official vault docker image running via ECS on EC2 container instances running Flatcar Linux

Vault server configuration template file with URLs and files modified to redact the domain, used in terraform:

storage "consul" {
  address = "${consul_http_address}"
  path    = "${vault_consul_path}"
}

service_registration "consul" {
  address     = "consul.internal:8200"
  scheme      = "https"
  tls_ca_cert = "/ssl/certs/ca.pem"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/ssl/certs/vault.internal.pem"
  tls_key_file  = "/ssl/private/vault.internal.pem"
}

ui = true

Additional context
VAULT_API_ADDR and VAULT_CLUSTER_ADDR are set in the environment. Here are the relevant log lines from startup of one of the servers IPs, IDs, and location information modified for purposes of redaction:

==> Vault server configuration:
AWS KMS KeyID: 01234567-890a-bcde-f012-34567890abcd
AWS KMS Region: down-south-1
Seal Type: awskms
Api Address: https://192.168.10.10:8200
Cgo: disabled
Cluster Address: https://192.168.10.10:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: info
Mlock: supported: true, enabled: true
Recovery Mode: false
Storage: consul (HA available)
Version: Vault v1.4.0
==> Vault server started! Log data will stream in below:

[ncabatoff]

Hi @inkblot,

Thanks for reporting this issue.

It sounds like you're having problems using the service_registration "consul" stanza in conjunction with storage "consul". Do the problems still exist if you remove the service_registration stanza altogether? There's no reason why you should have one:

The optional service_registration stanza configures Vault's mechanism for service registration. The service_registration stanza is designed for use cases where you would like to use a system like Consul for service discovery, but use a different system for the storage backend.

When Consul is configured as the storage backend, Vault implicitly uses Consul for service registration, so the service_registration stanza is not needed.

[inkblot]

I just reverted the configuration to check and it's the same both ways. I get the error [ERROR] service_registration.consul: error running service registration: redirect address must not be empty with the service_registration "consul" stanza removed as well as well as with it present.

I've been following #7887 since November in anticipation of migrating my storage off of consul. I've had service registration with consul working in vault versions up to 1.3.0. When I upgraded to 1.4.0 and registration stopped working I went ahead and added the service_registration "consul" stanza because I knew the feature was coming. I'd like to see the configuration work with the new stanza before I migrate the storage.

[ncabatoff]

I'm going to close this bug because it is fixed in #8976. We're due for a 1.4.2 release before too long which will include the fix.

[inkblot]

Thanks, looking forward to the release

[inkblot]

Confirmed fixed in 1.4.2