Vault CLI is unable to connect to Vault cluster using the Consul service address

[ykhemani]

Issue

Vault CLI is unable to connect to Vault cluster using the Consul service address.

The error I get when the Vault CLI tries to connect to the Vault cluster is the same regardless of the request. E.g. try logging in or try accessing a secret.

The Vault UI, a curl call to the API, and a Go client utilizing github.com/hashicorp/vault/api are able to connect to the Vault cluster utilizing the Consul service address.

To reproduce

0. Run Vault cluster with a Consul storage backend.

1. Run: VAULT_ADDR=<Vault Consul Service Address> vault <command>.

For example: VAULT_ADDR=https://vault.service.consul:8200 vault secrets list

2. See error.

Expected behavior

The expectation is that the Vault CLI would connect to the Vault cluster and return the response from the Vault cluster.

Environment

• Vault Server Version 1.4.0+prem.

• Vault CLI Version Vault v1.4.0+prem. Also tested with Vault OSS and older versions of Vault CLI.

• Consul storage backend uses Consul version Consul v1.7.2+ent.

• Server Operating System/Architecture: Ubuntu 18.04 / arm64.

Vault server configuration:

storage "consul" {
  address = "127.0.0.1:8501"
  scheme = "https"
  path = "vault/"
  tls_skip_verify = "true"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable = "false"
  tls_key_file = "/etc/ssl/private/privkey.pem"
  tls_cert_file = "/etc/ssl/certs/fullchain.pem"
  tls_min_version = "tls12"
}

api_addr = "https://ns3.home.example.com:8200"

disable_mlock="false"
disable_cache="false"
ui = "true"

max_lease_ttl="24h"
default_lease_ttl="1h"

#raw_storage_endpoint=true

cluster_name="hashi-vault"

insecure_tls="false"

plugin_directory="/data/vault/plugins"

telemetry {
  dogstatsd_addr = "localhost:8125"
  disable_hostname = true
}

Consul Server Configuration

datacenter           = "locust"
domain               = "consul.example.com"
node_name            = "ns3"
server               = true
bootstrap_expect     = 3
data_dir             = "/data/consul/data"

key_file             = "/etc/ssl/private/privkey.pem"
cert_file            = "/etc/ssl/certs/certificate.pem"
ca_file              = "/etc/ssl/certs/issuer.pem"
enable_script_checks = true
ports                = { "https" = 8501 }

addresses            = { "https" = "0.0.0.0" }
advertise_addr       = "192.168.1.4"
client_addr          = "0.0.0.0"
log_level            = "INFO"
ui                   = true

retry_join           = ["192.168.1.2","192.168.1.3","192.168.1.4"],

encrypt              = "<encrypt>"

Additional Detail

This is the error I get when Vault CLI tries to connect to Consul service address for the Vault cluster.

$ VAULT_ADDR=https://vault.service.consul.example.com:8200 vault kv get kv/demo/app2
Get https://vault.service.consul.example.com:8200/v1/sys/internal/ui/mounts/kv/demo/app2: dial tcp: lookup vault.service.consul.example.com on 192.168.1.2:53: no such host

DNS returns:

$ host vault.service.consul.example.com
vault.service.consul.example.com is an alias for ns2.home.example.com.

The Vault CLI is able to connect to the address of one of the Vault servers.

$ VAULT_ADDR=https://ns2.home.example.com:8200 vault kv get kv/demo/app2
====== Metadata ======
Key              Value
---              -----
created_time     2020-05-08T20:45:23.466680696Z
deletion_time    n/a
destroyed        false
version          1

======== Data ========
Key             Value
---             -----
api_token       E5145E36-F180-477A-BF56-E63DFC9D15BB
foo             bar
service_name    widget

The Vault CLI is also able to connect to the Consul node address of the Vault servers.

$ VAULT_ADDR=https://ns2.node.consul.example.com:8200 vault kv get kv/demo/app2
====== Metadata ======
Key              Value
---              -----
created_time     2020-05-08T20:45:23.466680696Z
deletion_time    n/a
destroyed        false
version          1

======== Data ========
Key             Value
---             -----
api_token       E5145E36-F180-477A-BF56-E63DFC9D15BB
foo             bar
service_name    widget

Curl is able to connect to the Consul service address for the Vault cluster.

curl -s \
  --header "X-Vault-Token: ${VAULT_TOKEN}" \
  https://vault.service.consul.example.com:8200/v1/kv/data/demo/app2 | jq -r .

An app written in go that utilizes the github.com/hashicorp/vault/api package is also able to talk to the Consul service address for the Vault cluster.

$ ./vgc -vault_path kv/data/demo/app2
INFO: vault_addr is https://vault.service.consul.example.com:8200
INFO: auth is token
INFO: vault_path is kv/data/demo/app2

Requested secret at path kv/data/demo/app2:
	api_token → E5145E36-F180-477A-BF56-E63DFC9D15BB
	foo → bar
	service_name → widget

Thanks so much!

[anth0d]

This seems to be a dupe of #9658 and more recently #12012

[ncabatoff]

I agree this is likely to be a dup as you say @anth0d. @ykhemani was your client running on macos?