Setting elasticsearch.sniff* settings in Kibana.yml causes authentication issues

[krisATelastic]

Kibana version:
7.2.0

Elasticsearch version:
7.2.0

Server OS version:
Centos 7.x

Browser version:
Chrome Version 75.0.3770.142 (Official Build) (64-bit)

Browser OS version:
Mac Mojave

Original install method (e.g. download page, yum, from source, etc.):
Yum from our repo

Describe the bug:
When setting the following keys in the kibana.yml configuration file;

elasticsearch.sniffInterval
elasticsearch.sniffOnConnectionFault
elasticsearch.sniffOnStart

... to anything other than false, the elasticsearch.username and elasticsearch.password seem to be ignored and revert to anonymous connections;

{"type":"log","@timestamp":"2019-07-30T02:55:36Z","tags":["status","plugin:xpack_main@7.2.0","error"],"pid":19934,"state":"red","message":"Status changed from yellow to red - [security_exception] missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

I also found that putting a typo in the elasticsearch.password to force a 403 would fail as expected however.

{"type":"log","@timestamp":"2019-07-30T03:06:05Z","tags":["status","plugin:xpack_main@7.2.0","error"],"pid":20139,"state":"red","message":"Status changed from yellow to red - [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

Steps to reproduce:

1. Build simple single node cluster with Kibana on same node
2. Enable xpack.security.enabled: true on Elasticsearch, run setup passwords
3. Configure kibana.yml to use the kibana user credentials in the elasticsearch.username and elasticsearch.password settings
4. Start Kibana, it will connect fine.
5. Stop Kibana and set elasticsearch.sniffInterval: 3000, elasticsearch.sniffOnConnectionFault: true and elasticsearch.sniffOnStart: true
6. Start Kibana, the above error should appear along with a long list of other authentication failures for missing authentication credentials

Work around
Do not use these settings, either drop them from the configuration or set all 3 to false.

[elasticmachine]

Pinging @elastic/kibana-operations

[azasypkin]

@elastic/kibana-operations I've tagged you on this issue since you implemented this feature in #21928, but let me know if the issue is related to the authentication itself that Security Team should normally handle.

[krisATelastic]

I can also confirm that i've reproduced this on version 7.3.1 also.

With elasticsearch.sniffInterval and elasticsearch.sniffOnStart settings set, this is returned in the Kibana log;

Sep 5 00:24:31 krisr-kibanatest01 kibana[24825]: {"type":"log","@timestamp":"2019-09-05T00:24:31Z","tags":["license","warning","xpack"],"pid":24825,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

[tylersmalley]

@jbudz do you have some cycles to look into this?

[jbudz]

Yeah, I'll have to take a closer look at client lib but this came up in testing. The sniff settings will pull cluster information from elasticsearch, but it isn't necessarily the same endpoints and information kibana uses to connect (proxies and so on). I didn't find it very useful in practice, but there's room for improvement with auth here.

[benatbermejo]

Reproduced on version 7.5. When I change elasticsearch.sniffOnStart to true, I get 401 error trying to get license information on startup.

[xiu]

We have the same issue in 7.4.2

[douniwan5788]

same issue in 7.6.2
Thank God! finally found the reason

[joshdover]

@elastic/kibana-platform do we know if this problem also exists with the new client?

[pgayvallet]

@delvedor ^ Maybe you will be able to answer this one?

[ansalon282]

i get the same error on newer version 7.9.3 and 7.10.0

[delvedor]

@pgayvallet sorry for the delay. This should not happen with the client, as the sniff logic is using the same code that every other request uses, so the authentication should work as well.

[xiaobao623]

I also got same issue in kibana-7.9.3 after adds below 3lines.

elasticsearch.sniffInterval: 300000
elasticsearch.sniffOnStart: true
elasticsearch.sniffOnConnectionFault: true

[pgayvallet]

@jbudz I know it's been a while now, but did you get to investigate further?

[jbudz]

It's still an issue, possibly pending the full migration via #83910. Same authorization failure - nodes are added to the cluster list but requests don't pass kibana.yml credentials. Errors manifest as Kibana loading but anything hitting es fails to load due to the license check failing.

[artem-shestakov]

One parameter is enought - elasticsearch.sniffInterval After that time:
{"statusCode":401,"error":"Unauthorized","message":"missing authentication credentials for REST request [/.kibana/_doc/space%3Adefault]: security_exception"}

[pgayvallet]

nodes are added to the cluster list but requests don't pass kibana.yml credentials

Oh I see, that's what is happening here. Nodes are added, but the way we're adding auth is not propagated, which makes sense, in a way, as for the legacy client we were adding auth to the individual nodes.

kibana/src/core/server/elasticsearch/legacy/elasticsearch_client_config.ts

Lines 129 to 131 in 4584a8b

	if (needsAuth) {
	host.auth = `${config.username}:${config.password}`;
	}

It should be fixed when completing the migration to the new client then, as we are defining the auth in a more global way

kibana/src/core/server/elasticsearch/client/client_config.ts

Lines 77 to 82 in 4584a8b

	if (config.username && config.password && !scoped) {
	clientOptions.auth = {
	username: config.username,
	password: config.password,
	};
	}

@delvedor does that sound correct? The nodes added by sniffing requests are correctly using the top-level defined auth (ClientOptions.auth), right?

Do you know if, for the legacy client, using ConfigOptions.httpAuth instead of ConfigOptions.hosts.auth would allow sniffed node to also correct use the http auth?

[pgayvallet]

Fixed starting in versions 7.12 and 7.11.2