Only connect to Elasticsearch instances with the same version or newer

[kvch]

What does this PR do?

This PR adds a new callback to all Elasticsearch connections to check if the version is at least the same as the Beat. The check can be disabled by setting the new option output.elasticsearch.allow_older_versions to true.

Why is it important?

By restricting the Elasticsearch versions we can provide optimal experience.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #29264

[mergify]

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-01-12T16:28:44.619+0000

• Duration: 154 min 8 sec

• Commit: f331836

Test stats 🧪

Test	Results
Failed	0
Passed	49893
Skipped	4442
Total	54335

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ruflin]

@kvch Is my understanding correct that we can target this against 8.1 as 8.0 does not talk to previous major versions anyways?

[kvch]

Is my understanding correct that we can target this against 8.1 as 8.0 does not talk to previous major versions anyways?

We target this against 8.1. In 8.0, if someone tries to connect to an older version, we just emit a warning: #29723

[kvch]

/test filebeat

[ruflin]

@nimarezainia Can we make sure this is also clearly shown in our compatibility matrix?

@kvch Based on your comment, someone could run Beats 8.0 vs ES 7.13 and the user would get a warning. Do we agree that we don't support this?

[kvch]

@kvch Based on your comment, someone could run Beats 8.0 vs ES 7.13 and the user would get a warning. Do we agree that we don't support this?

Yes and yes.

[mgevans-5]

Please confirm this does or does not impact Logstash as the PR only mentions connections to Elasticsearch

[cmacknz]

Please confirm this does or does not impact Logstash as the PR only mentions connections to Elasticsearch

The change only affects Elasticsearch. The only code change here is to the beats Elasticsearch client.