Support for multiple Elasticsearch nodes and sniffing #21928

jbudz · 2018-08-13T15:39:53Z

This adds support for node sniffing. The elasticsearch-js client supports this natively, but we create proxies that bypass the client. Two routes are used for searching and these are rewritten to use the client.

The last is used by Console, and not covered in this pull request - the first host will be used.

Configuration changes, ref:

elasticsearch.url (string) -> elasticsearch.hosts [(string)]
elasticsearch.sniffOnStart
elasticsearch.sniffInterval
elasticsearch.sniffOnConnectionFault

test/api_integration/apis/elasticsearch/index.js

x-pack/plugins/monitoring/config.js

src/core_plugins/elasticsearch/index.js

src/core_plugins/elasticsearch/lib/create_proxy.js

pickypg

Monitoring changes LGTM. Probably worth getting a once-over from @elastic/stack-monitoring to ensure they're aware / test it.

chrisronline · 2018-12-17T20:38:48Z

Can someone summarize the meaning of this? Do we mean sniffing ES nodes other than the single node Kibana knows about? Or is this something else?

elasticmachine · 2018-12-17T20:57:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

jbudz · 2018-12-18T16:08:20Z

You got it. Kibana (if these settings are turned on) will look for other nodes in the cluster using elasticsearch endpoints and add them to its list of hosts. If any node goes down it will switch to a different node and continue operating.

* Node sniffing * update monitoring * cleaning * remove tests * rewrite tests using the proxy * fix monitoring mocha tests * fix payload * return full error * default interval false * add sniff settings to monitoring * add docs for sniff settings * re-add index search integration test * catch parse error * capital * fix merge * hapi upgrade * return * pass by prefix * rm empty test fille * split error handling * fix merge * update recent elasticsearch.url references * prettier vs eslint * transport.request, include query params * mega comment * revert elasticsearchUrl * safe hosts * more docs es.url references * remove unused ml/esServerUrl * log if both set

…27431) * Support for multiple Elasticsearch nodes and sniffing (#21928) * Node sniffing * update monitoring * cleaning * remove tests * rewrite tests using the proxy * fix monitoring mocha tests * fix payload * return full error * default interval false * add sniff settings to monitoring * add docs for sniff settings * re-add index search integration test * catch parse error * capital * fix merge * hapi upgrade * return * pass by prefix * rm empty test fille * split error handling * fix merge * update recent elasticsearch.url references * prettier vs eslint * transport.request, include query params * mega comment * revert elasticsearchUrl * safe hosts * more docs es.url references * remove unused ml/esServerUrl * log if both set * tribe test * update remaining url references

DaveyDevOps · 2019-03-22T16:23:21Z

docs/setup/settings.asciidoc

+
+`elasticsearch.sniffOnStart:`:: *Default: false* Attempt to find other Elasticsearch nodes on startup.
+
+`elasticsearch.sniffOnConectionFault:`:: *Default: false* Update the list of Elasticsearch nodes immediately following a connection fault.


Hello, looks like some bits fell out. Connection is missing an 'n' in the setting name.

…Ref: elastic/kibana#21928

* 6.x: Rework index page and version matrix New EOL Replcae .url by .hosts in all files. Plus fix English typos. Elasticsearch .url config option was deprecated. Use .hosts instead. Ref: elastic/kibana#21928 Add note about "elasticsearch.ssl.alwaysPresentCertificate: true" for Kibana client cert authentication

* 6.x: 7x-35 is latest [noindex] Rework index page and version matrix New EOL Replcae .url by .hosts in all files. Plus fix English typos. Elasticsearch .url config option was deprecated. Use .hosts instead. Ref: elastic/kibana#21928 Add note about "elasticsearch.ssl.alwaysPresentCertificate: true" for Kibana client cert authentication

* 6.x: 7x-35 is latest [noindex] Rework index page and version matrix New EOL Replcae .url by .hosts in all files. Plus fix English typos. Elasticsearch .url config option was deprecated. Use .hosts instead. Ref: elastic/kibana#21928 Add note about "elasticsearch.ssl.alwaysPresentCertificate: true" for Kibana client cert authentication # Conflicts: # _config.yml

* 6.x: 7x-35 is latest [noindex] Rework index page and version matrix New EOL Replcae .url by .hosts in all files. Plus fix English typos. Elasticsearch .url config option was deprecated. Use .hosts instead. Ref: elastic/kibana#21928 Add note about "elasticsearch.ssl.alwaysPresentCertificate: true" for Kibana client cert authentication

* 6.x: Rework index page and version matrix New EOL Replcae .url by .hosts in all files. Plus fix English typos. Elasticsearch .url config option was deprecated. Use .hosts instead. Ref: elastic/kibana#21928 Add note about "elasticsearch.ssl.alwaysPresentCertificate: true" for Kibana client cert authentication

jbudz added WIP Work in progress Team:Operations Team label for Operations Team labels Aug 13, 2018

This comment has been minimized.

Sign in to view

jbudz force-pushed the node-sniffing branch 2 times, most recently from ecd890b to f2776a4 Compare October 16, 2018 13:47

This comment has been minimized.

Sign in to view

jbudz added 5 commits October 16, 2018 09:36

Node sniffing

bd542c2

update monitoring

9bb693f

cleaning

1e021a1

remove tests

9c76c88

rewrite tests using the proxy

55743a0

jbudz force-pushed the node-sniffing branch from 853be76 to 55743a0 Compare October 16, 2018 14:38

This comment has been minimized.

Sign in to view

fix monitoring mocha tests

9d33df9

jbudz commented Oct 16, 2018

View reviewed changes

test/api_integration/apis/elasticsearch/index.js Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

jbudz added 2 commits October 16, 2018 13:01

fix payload

10d3933

Merge branch 'master' into node-sniffing

99b4927

This comment has been minimized.

Sign in to view

return full error

8a37e35

jbudz commented Oct 16, 2018

View reviewed changes

x-pack/plugins/monitoring/config.js Outdated Show resolved Hide resolved

jbudz commented Oct 16, 2018

View reviewed changes

src/core_plugins/elasticsearch/index.js Outdated Show resolved Hide resolved

jbudz commented Oct 16, 2018

View reviewed changes

src/core_plugins/elasticsearch/index.js Outdated Show resolved Hide resolved

jbudz commented Oct 16, 2018

View reviewed changes

src/core_plugins/elasticsearch/lib/create_proxy.js Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

jbudz added 2 commits October 16, 2018 16:51

default interval false

1207572

add sniff settings to monitoring

2898bba

jbudz added 2 commits December 17, 2018 14:04

remove unused ml/esServerUrl

070ecd6

log if both set

2bf1e65

pickypg approved these changes Dec 17, 2018

View reviewed changes

jbudz merged commit b5b64f4 into elastic:master Dec 18, 2018

jbudz added v7.0.0 v6.6.0 backport pending labels Dec 18, 2018

jbudz removed the backport pending label Dec 19, 2018

jbudz mentioned this pull request Jan 7, 2019

Support multiple Elasticsearch nodes and sniffing #214

Closed

This was referenced Feb 4, 2019

[DOCS] Clarify deprecation of Kibana settings #29979

Closed

[DOCS] Deprecate elasticsearch.url #29982

Merged

[DOCS] Deprecate monitor settings #30004

Merged

jbudz mentioned this pull request Mar 19, 2019

[stable/kibana] elasticsearch.url is deprecated helm/charts#12378

Closed

DaveyDevOps reviewed Mar 22, 2019

View reviewed changes

sergibondarenko mentioned this pull request Apr 17, 2019

Elasticsearch .url config option was deprecated floragunncom/search-guard-docs#67

Merged

floragunn pushed a commit to floragunncom/search-guard-docs that referenced this pull request Apr 29, 2019

Elasticsearch .url config option was deprecated. Use .hosts instead. …

1f1cab0

…Ref: elastic/kibana#21928

azasypkin mentioned this pull request Aug 15, 2019

Setting elasticsearch.sniff* settings in Kibana.yml causes authentication issues #42224

Closed

russcam mentioned this pull request Feb 12, 2020

LoadBalancer kills Logstash connections elastic/azure-marketplace#249

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for multiple Elasticsearch nodes and sniffing #21928

Support for multiple Elasticsearch nodes and sniffing #21928

jbudz commented Aug 13, 2018 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pickypg left a comment

chrisronline commented Dec 17, 2018

elasticmachine commented Dec 17, 2018

jbudz commented Dec 18, 2018

DaveyDevOps Mar 22, 2019


		`elasticsearch.sniffOnStart:`:: Default: false Attempt to find other Elasticsearch nodes on startup.

		`elasticsearch.sniffOnConectionFault:`:: Default: false Update the list of Elasticsearch nodes immediately following a connection fault.

Support for multiple Elasticsearch nodes and sniffing #21928

Support for multiple Elasticsearch nodes and sniffing #21928

Conversation

jbudz commented Aug 13, 2018 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pickypg left a comment

Choose a reason for hiding this comment

chrisronline commented Dec 17, 2018

elasticmachine commented Dec 17, 2018

💚 Build Succeeded

jbudz commented Dec 18, 2018

DaveyDevOps Mar 22, 2019

Choose a reason for hiding this comment

jbudz commented Aug 13, 2018 •

edited

Loading