[discuss] new elasticsearch client version management

[pgayvallet]

Initial discussion here: #69905 (comment)

In #35508 we are going to introduce the new elastic search (https://github.com/elastic/elasticsearch-js)

This issue is to discuss the possible option to keep each kibana branches up to date with the expected client's version.

The main problem to solve is that the client is no longer (at least officially) compatible between versions. Meaning that we shouldn't theoretically be using a 7.x client on our master branche, as it targets 8.0, and we shouldn't have a 8.0 client on our 7.x branches.

Another thing to take into consideration is that elastic/elasticsearch-js follow the stack releases. That means that during the development period of a given version, we can't pin the library to the same version.

From the discussion we had with @delvedor and @restrry, the possible options could be

(please edit/fix/improve)

use the last released version of the client on kibana master branch

For example today, we would be using the 7.8.0 version of the client on both kibana master and 7.x branches, as it's the last version of the stack released to this day (and therefor the last pinned version of the client)

Pros:

• we got the same version of the client on our master and 7.x branches, which should ease backporting a lot

Cons:

• we can't use any API added in higher versions on master without using the transport.request workaround.
• In case of breaking changes between majors, we would have an incompatible version on master

use master version of the client on kibana master branch

We would be using the master/head version of the client on kibana master, and pinned version on our 7.x branch. When a branch is cut from 7.x, we then bump the client version on 7.x. master is always using the master version of the client.

Pros:

• Better client targeted version
• Allow to use the last APIs on master

Cons:

• This doesn't solve the in-development version API mismatch: Today our 7.x branch is targeting the 7.9 version, however the last version of the client is 7.8, as 7.9 is not, yet, released. As we can't use client's master on 7.x, we would still be forced to rely on transport.request for any potential 7.9 ES APIs (at least until FF when the client's version is released, but that would then mean to start developing with transport.request and then modify the feature after FF, which doesn't seems like a great idea)
• May be wrong, but I'm a little afraid on API/Types/changes on the client itself between versions. That could increase divergence between master and 7.x and cause some tedious backporting in some cases.

Questions:

• if we go that way, to which extend could we automate the version bumping + api generation on new branch / releases?

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[delvedor]

The versioning strategy of the client is slightly different, let me summarize it :)

• The client master branch follows the ES master branch.
• The client 7.x branch follows the ES 7.x branch (which currently is 7.9).

Once we go in feature freeze and (in this case) the 7.9 branch is cut from 7.x in ES, the client does the same, so there will be a 7.9 branch in the client as well, while 7.x will continue to follow 7.x (which will be 7.10).

As soon as we are in feature freeze, the client is released on npm as rc with the next tag, so it can be installed with npm install @elastic/elasticsearch@next.
I don't think it should be a problem installing from GitHub during the development phase, as you can pin a dependency to a specific branch/commit. Anyhow, if you prefer to have something published on npm during the development phase, we can figure out a strategy that works well for us.

Finally, I would like to mention that the client follows semver and there will never be a breaking change between minor releases, the only thing that might change, but there must be a very good reason, are the helpers and the type definitions, as both of them are still in experimental phase.
The type definitions of the client itself are stable, while we are working on improving the request/response types.

[pgayvallet]

I don't think it should be a problem installing from GitHub during the development phase

For our master branch, now that locks are a thing, pointing to @master may not be an issue (even if AFAIK we are not doing that with any other library)

For our 7.x branch, pointing to @next may be more difficult, as:

• we would need to change the version when creating the next release branch (i.e 7.9) from 7.x, so this means adapting the release script/process
• the @elastic/elasticsearch:7.9.0 would be REQUIRED to be release before we create Kibana 7.9 branch, else we would point to a non-existing version.
• even if it should, we can't be 100% sure that when switching, @next and v7.9.0 will be the exact same version, which could cause failures when creating the Kibana release branch

[mshustov]

we would need to change the version when creating the next release branch (i.e 7.9) from 7.x, so this means adapting the release script/process
the @elastic/elasticsearch:7.9.0 would be REQUIRED to be release before we create Kibana 7.9 branch, else we would point to a non-existing version.

@elastic/kibana-operations would it be a problem to automate the dependency version update? Do we bump version manually?

@delvedor Do you release a new version for every major and minor ES releases? But not for patch?

even if it should, we can't be 100% sure that when switching, @next and v7.9.0 will be the exact same version, which could cause failures when creating the Kibana release branch

sounds like a proper use-case for mono-repo 😅

[delvedor]

@delvedor Do you release a new version for every major and minor ES releases? But not for patch?

Currently, the client strictly follows major and minor releases, but as soon as the clients will be a part of the unified release, also patches will be aligned.

even if it should, we can't be 100% sure that when switching, @next and v7.9.0 will be the exact same version, which could cause failures when creating the Kibana release branch

Not sure I follow, can you elaborate? If you install from a tag (@next for instance) you will get the version that is published behind that tag in the package.json, for example:

npm i @elastic/elasticsearch@next

{
  "dependencies": {
    "@elastic/elasticsearch": "^7.8.0-rc.1"
  }
}

For our 7.x branch, pointing to @next may be more difficult

Yes, it will. We must figure out a process that addresses all the issues.

the @elastic/elasticsearch:7.9.0 would be REQUIRED to be release before we create Kibana 7.9 branch, else we would point to a non-existing version.

Unfortunately, I don't think this could happen, I cannot release before the stack GA nor before FF.
But I don't think it will be a problem shipping an rc version in Kibana during FF, as the client will only do bug fixes, but the API will no longer be touched. I remember @joshdover and I already had a small discussion about this in private.

[costin]

Folks, I don't think you realize that by using @ elastic / elasticsearch you notified all members of said repository.
And through @ next : https://github.com/next

[tylersmalley]

Not sure I follow, can you elaborate? If you install from a tag (@next for instance) you will get the version that is published behind that tag in the package.json, for example:

This won't work because it's pinned to a specific version in the lock file.

Today I mentioned that we could utilize Renovate to assist with these bumps, however that is currently only configured to monitor the master branch. Adding additional branches for a single dependency would ultimately result in more noise. Currently the EUI team has a similar approach where after releasing a new version, they create a PR to update Kibana.

I would really like to avoid using a Git reference for this dependency in master. I have not verified, but I am sure it wouldn't leverage caching to the degree that NPM packages do, adding needless time to the dependency installation/resolution time.

Would a viable option be to publish beta releases to be used ahead of actual stack releases?

• master would define elasticsearch-js: ~8.0 and would use the published 8.0.0-XXX (using a bc, beta, or some qualifier).
• 7.x branch is currently 7.9 which would have elasticsearch-js: ~7.9 using a published 7.9.0-XXX

The main issue I see with this is that for every release, each tracked branch would have to have it's dependency bumped. It's possible we could write a script to assist with this. Or, it would be part of the process when a new version of elasticsearch-js is released, beta or otherwise.

[delvedor]

Currently the EUI team has a similar approach where after releasing a new version, they create a PR to update Kibana.

I spoke with @chandlerprall and it's not exactly the same, unfortunately, as they don't need to follow the stack versioning strictly.

Would a viable option be to publish beta releases to be used ahead of actual stack releases?

We could do this by hand, it should not be an issue. How do you feel about creating a package on npm just for Kibana? I would like to avoid polluting the official client on npm with too many prereleases, otherwise, the UX for developers choosing a version would be a nightmare.
We could publish a separate package, for example @elastic/elasticsearch-kibana or @elastic/elasticsearch-prebuild, where we can publish as many prereleases as we need.

Can our current automation bump versions even if they are prereleases?

[pgayvallet]

Added Team:Operations label as it seems they are now directly implicated with this issue.

I would like to avoid polluting the official client on npm with too many prereleases, otherwise, the UX for developers choosing a version would be a nightmare

I can understand that, however with npm version selection syntax (version: '~7.8.0' or version: '^7.8.0'), I felt like this wouldn't have much impact for developers (also I think we would only need one prerelease for each stack release). But in the end, this decision is only yours to take, so if you say this is a no-go, we can directly exclude this option.

We could publish a separate package, for example elastic/elasticsearch-kibana, where we can publish as many prereleases as we need.

I guess that may be a solution, however I'm afraid of a few things:

• That would force us to import the client from another package than elastic/elasticsearch (Even if I can't find why this would really be blocker in any way)
• It feels like it would be more work to maintain this secondary package. We won't be able to 'switch' from one package to another during dev cycle, so every release of elastic/elasticsearch would have to be 'mirrored' to this elastic/elasticsearch-kibana, in addition to any dev/rc/beta version we will need for the development branches. Also, what would exactly be the workflow to publish versions to this package?

Can our current automation bump versions even if they are prereleases?

I will let @tylersmalley answer this one.

[delvedor]

I felt like this wouldn't have much impact for developers

In a "normal" module, where only the latest version is maintained, yes. But given that the client needs to support 2/3 release lines at the same time, developers might find it useful to visit the npm versions page.

That would force us to import the client from another package than elastic/elasticsearch (Even if I can't find why this would really be blocker in any way)

It should not be a problem, you just need to rename the import to import { ... } from '@elastic/elasticsearch-kibana'. Since the types are directly shipped with the client, ts will not complain.

It feels like it would be more work to maintain this secondary package. We won't be able to 'switch' from one package to another during dev cycle

You don't need to. The idea is to have a "public" package, which is @elastic/elasticsearch that it the one used by everyone, and @elastic/elasticsearch-kibana used by kibana only. The codebase is the same, I only publish it twice in two different namespaces, where the kibana one has all the prereleases needed to make this work. For example, currently 7.x is 7.9, so we would have 7.9.0-alpha.${number}.

Also, what would exactly be the workflow to publish versions to this package?

I can easily write a script to handle this process, given that the codebase is the same, it's just a matter of renaming and publishing.

[mshustov]

You don't need to. The idea is to have a "public" package, which is @elastic/elasticsearch that it the one used by everyone, and @elastic/elasticsearch-kibana used by kibana only. The codebase is the same, I only publish it twice in two different namespaces, where the kibana one has all the prereleases needed to make this work. For example, currently 7.x is 7.9, so we would have 7.9.0-alpha.${number}.

Ok for me. @elastic/kibana-operations @tylersmalley any objections?
these 2 questions aren't answered yet:

Can our current automation bump versions even if they are prereleases?

would it be a problem to automate the dependency version update? Do we bump version manually?

[tylersmalley]

I am not opposed to creating elasticsearch-kibana for pre-releases. It's something that we could easily change in the future.

Can our current automation bump versions even if they are prereleases?

Currently version bumps are done by hand by us. If the package is published ahead of time, we can handle the bump at the same time. We have an open issue to automate this here: #25769

[delvedor]

I am not opposed to creating elasticsearch-kibana for pre-releases. It's something that we could easily change in the future.

@tylersmalley actually I was thinking that you should not use the elasticsearch-{kibana/prerelase/nightly/prebuild} only for pre-releases, but just depend on it for both pre-releases and normal relases.
I'll take care of keep it updated on npm so you don't need to go back and forth between the different packages.

[Mpdreamz]

Naming is hard and subjective but I would prefer if kibana is not in the package name. Will be confusing for folks who scour NPM and mistake it for e.g a kibana API client.

elasticsearch-canary/prerelease would be my preference as it clearly signals what sets it apart from the regular client.

Currently version bumps are done by hand by us. If the package is published ahead of time,

We are looking to tie the releases of the clients into the unified release. So therefor unlikely to be available ahead of time since. I reckon swapping packages at version bump/tagging is in general not something you'd want, given the other packages is the one that went through all the testing.

For example, currently 7.x is 7.9, so we would have 7.9.0-alpha.${number}.

Nitpick but I would not use alpha/beta/rc as prerelease monikers for this canary build. Since the stack will do alpha/beta/rc releases during majors this could end up being confusing. If these packages always ship as 7.9.0-canary.${number}. ${number} could also be the date e.g 20180906T170958. Having a distinct moniker helps separate it from the mainline versioning.

[joshdover]

actually I was thinking that you should not use the elasticsearch-{kibana/prerelase/nightly/prebuild} only for pre-releases, but just depend on it for both pre-releases and normal relases.

@delvedor any updates here on this?

[delvedor]

@joshdover I'm waiting for you folks to confirm that this approach works for you :)

To recap:
I will publish on npm two packages:

• @elastic/elasticsearch
• @elastic/elasticsearch-canary

The first one will be used by everyone, while Kibana will depend on the second one, both for pre and normal releases.
The only difference between the two is that the first one will follow the stack release schedule, while the second one will be released as soon as there are new changes.

[mshustov]

@delvedor It seems we have an agreement on this.
Btw when 7.10 is going to be released? Our 7.10 branch still points to 7.9.1 https://github.com/elastic/kibana/blob/7.10/package.json#L116

[mshustov]

Currently version bumps are done by hand by us. If the package is published ahead of time, we can handle the bump at the same time. We have an open issue to automate this here: #25769

@tylersmalley Is there a doc describing all the steps to do when bumping a version? 7.10 still uses @elastic/elasticsearch: 7.10-rc.1 https://github.com/elastic/kibana/blob/7.10/package.json#L118
Can we automate version update?

[tylersmalley]

@restrry @delvedor, I have updated our doc and notified @mistic who has been doing our version bumps of this addition to the process. Apologies for not have already started it, I missed this as being settled, my bad.

Is there anything we can do to ensure that the version has been created ahead of time? If it has not been created, is this something we could also do as part of this process?

[mistic]

I've read the comments on the issue but it was not clear to me what was our chosen strategy to handle this (sorry if I miss os misread something 😃 ). I think in my head it makes sense if we can be always in sync with the package across any tracked branch. I strategy that came up into my mind was trying to use @semantic-release to automate the release of the package as much as we can and rely in tags to always be able to depend on the most update changes of the package for each tracked branch. Something like:

package@next -> points to master
package@7.10.0 or package -> points to last public published version
package@7.10-next -> points to 7.10 branch
package@7.x  -> points to 7.11 branch

Once 8.0 got release @next will point to 9.0 and all of the tags should be updated.

@tylersmalley @restrry @delvedor @pgayvallet what do you think about this? It probably can be improved but in my head it makes sense and I think it will make our life easier to keep our kibana package.json in sync with the less amount of effort.

[pgayvallet]

package@next -> points to master

You can't have a npm tag point to a GH branch, if that what you meant?

[delvedor]

Hello! With elastic/elasticsearch-js#1338 the client will start publishing two packages:

• @elastic/elasticsearch
• @elastic/elasticsearch-canary

The first one is the "normal" distribution and will follow stack releases. The second one will be released as soon as there are new changes in the client, such as new features o new supported APIs.
Kibana will always depend on @elastic/elasticsearch-canary, both for development and releases, the canary package will release the stable version of each release as well.
The branches and dependencies should be organized in this way (as of 9 Dec 2020):

kibana master: `@elastic/elasticsearch-canary@8.0.0-canary-{incremental integer}`
kibana 7.x: `@elastic/elasticsearch-canary@7.11.0-canary-{incremental integer}`
kibana 7.10: `@elastic/elasticsearch-canary@7.10.x`

One of the main reasons why we need the canary package is that there is no other way for Kibana to get the new ES features before feature freeze.

@tylersmalley @mistic does this look ok for you?

[mistic]

@pgayvallet when I said pointing to master I meant @next will always reflect the most update changes we have on that branch which will imply the integration of an automatic release tool like @semantic-release. 😃

@delvedor I'm okay with that approach. Just one question: if kibana will always depend on @elastic/elasticsearch-canary, even on releases, where will @elastic/elasticsearch be used?

[delvedor]

Just one question: if kibana will always depend on @elastic/elasticsearch-canary, even on releases, where will @elastic/elasticsearch be used?

@mistic @elastic/elasticsearch will not be used at all. Try to sync the release is really annoying, my plan is to remove completely this issue by making Kibana depend only on the canary package even for releases, as the canary package will publish stable versions as well. In this way, your work on the automation side should be easier.

[mistic]

@delvedor got it, I'm okay with it. \cc @tylersmalley

[delvedor]

The canary package is available on npm 🚀
https://www.npmjs.com/package/@elastic/elasticsearch-canary?activeTab=versions

[delvedor]

@elastic/kibana-operations there are two new versions of the canary package. How will the upgrade be handled? Is it possible to automate it?

[spalger]

@delvedor I'll put up a PR now to get the migration to 8.0.0-canary.2 started, I believe we'll need to address #88010 as part of that upgrade.

[spalger]

Updated our renovate config to automatically submit PRs on new @elastic/elasticsearch versions independently for:

• master Update dependency @elastic/elasticsearch to ^8.0.0-canary.3 (master) #91706
• 7.x Update dependency @elastic/elasticsearch to ^7.12.0-canary.3 (7.x) - autoclosed #91716

[jbudz]

Are we good here? We've agreed on canary versions and have implemented renovate configuration setup, I think that covers the discussion so far.