Lower CI Java Version to Java 11

[nknize]

Java 11 is an LTS release, Java 14 is not. Elastic really shouldn't have made the minimum java version 14 prematurely. Lucene doesn't even require minimum 14. The community opened a PR to lower the minimum required java version back to 11 but it is stalled until CI can be lowered to match (otherwise CI will fail). The PR has stalled now for almost two months and the contributor has long given up. This is unfortunate and really quite unacceptable. Maybe we can get this taken care of to unblock that PR.

Update: the remaining work is as follows. OpenSearch has CI defined in a private freestyle project that runs gradle check on OpenSearch#main, branches and PRs. Extract that into Jenkinsfile that is committed to the OpenSearch repo, similar to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/Jenkinsfile.

[dblock]

This issue belongs in https://github.com/opensearch-project/opensearch-build, and should be assigned to @bbarani. Can someone please move it?

[peterzhuamazon]

Hi @nknize as of now the Infra Team did not add any JDK into the release artifact.
In fact, the JDK 14 is bundled with the core/min artifacts during the Gradle Assemble process.
From Infra Side, we only install the plugins and re-tar artifacts as bundle.

If this is something that can be resolved in gradle assemble that would be great.
If this is a hard target to fix in core, we can see what we can do in the bundle process.

Thanks.

[dblock]

@peterzhuamazon Are you saying the version of Java that the build is run with isn't set to 14 in Jenkins?

[nknize]

@peterzhuamazon Are you saying the version of Java that the build is run with isn't set to 14 in Jenkins?

JAVA{8-15}_HOME is required to be set to build. So the build has to be setting those. Also, jdk is bundled and can be specified with: -Druntime.java=14. Is the CI runtime specifying this?

[peterzhuamazon]

@peterzhuamazon Are you saying the version of Java that the build is run with isn't set to 14 in Jenkins?

We set the build to 14 yes but that was purely due to JDK14 has been used for engine throughout ODFE.
I do not have knowledge on how JDK is being taken care during the assemble process.
Are you saying the gradle assemble take JDK and bundle the specific version based on the JDK env during build?

[nknize]

We set the build to 14 yes but that was purely due to JDK14 has been used for engine throughout ODFE.

That's because it was the min java requirement of Elasticsearch. Which is being changed by the community contributed PR.

Can we go ahead and merge the PR and have you set the build to 11?

[camerski]

That PR needs to be merged before we can change the CI setup anyway...if we change CI first, then everything will break. That PR changes the minimum JDK version to 11. If CI is still running 14 then that shouldn't break. We'll flip CI to use JDK 11 after the PR is merged.

I recommend merging the PR now. Worst case, we just revert the change.

[dblock]

I've done a rebase and am finishing this up in opensearch-project/OpenSearch#940

[dblock]

@camerski that PR was merged, let's flip CI for main and see what happens?

[camerski]

I'll add it to our backlog

[camerski]

Successfully built OpenSearch/main using Java 11 in our CI tool.

Resolving.

[dblock]

Successfully built OpenSearch/main using Java 11 in our CI tool.

Resolving.

Which branches has this been enabled for? I think we should do it for main and 1.1.
Do you have a link to a log of a successful build?

[dblock]

I am seeing this in the log of this PR on main.

  JDK Version           : 14 (AdoptOpenJDK)

Reopening.

[dblock]

@camerski

[camerski]

Right now we only build the 1.0 branch, and that branch does not have the JDK-11 requirement backported yet. We put the CI back to JDK 14 to match what is in 1.0.

Once the backport is done, let us know and we'll drop the CI JDK version again.

[camerski]

You are correct that this issue should have been re-opened at the time we bumped the CI back to JDK 14.

[dblock]

Right now we only build the 1.0 branch, and that branch does not have the JDK-11 requirement backported yet. We put the CI back to JDK 14 to match what is in 1.0.

I don't think I follow. This is what I'd expect to happen:

• Main should be building 11, including gradle checks.
• 1.x should be building 11, including gradle checks.
• 1.0 should be building 14, including gradle checks.

Once the backport is done, let us know and we'll drop the CI JDK version again.

We are not going to backport 14 -> 11 to 1.0, that should continue building with 14.

Each branch needs to be able to say which JDK to use. Furthermore, we will be wanting to run integration tests on multiple JDKs for each project across multiple branches and they need to be able to express that.

[camerski]

The daily snapshots run in CI are currently only building 1.0.

Once we set up daily snapshots for the other branches we can use JDK 11 for those. There is no ETA for setting up daily snapshots for main and 1.x.

Agree that each branch should be able to say which JDK to use. Any ideas for how best to communicate that? Ideally it would be something checked in to the repository so the CI can just check out the branch, discover the right JDK, and go from there.

[dblock]

The daily snapshots run in CI are currently only building 1.0.

What about all the other CI on main and 1.x? CI that runs on PRs? Gradle checks? I want those using 11.

Once we set up daily snapshots for the other branches we can use JDK 11 for those. There is no ETA for setting up daily snapshots for main and 1.x.

OK.

Agree that each branch should be able to say which JDK to use. Any ideas for how best to communicate that? Ideally it would be something checked in to the repository so the CI can just check out the branch, discover the right JDK, and go from there.

We have https://github.com/opensearch-project/OpenSearch/blob/main/.ci/java-versions.properties that declares the version that should be used for building and running tests. Does this work?

[camerski]

Yeah, we can probably make that work. I'll add it to the backlog.

[dblock]

Now that we have a build workflow running I think we need the following:

1. Create child issues in each repo to lower their GHA CI to JDK 11.
2. Update Jenkins workflow in this repo to use JDK11.
3. Check that everything is using 11.

[reta]

@camerski could you please help me with CI images?

In Jenkinsfile we have right now image 'opensearchstaging/ci-runner:centos7-x64-arm64-jdk14-node10.24.1-cypress6.9.1-20211005 (which is fine for 1.x). Since we lowered 2.x requirements to JDK-11 and also added JDK-17 support (see please opensearch-project/OpenSearch#1351), it would be great to have respective images for jdk11 and jdk17 available. Once they are baked, we could move on with the build changes.

If you need any hints what JDK distributions to use, https://adoptium.net/ would make it.
Thank you.

[dblock]

Cameron has changed teams, @peterzhuamazon and @peternied can help. I think we should discuss #732 first?

[reta]

@dblock haha, Pandora box, definitely makes sense to split runners first, will continue the discussion on #732, thank you!

[dblock]

After opensearch-project/OpenSearch#1368, the gradle check is still using Java 14. This comes hardcoded from the infrastructure that runs these. We need it to start picking up Java version from https://github.com/opensearch-project/OpenSearch/blob/main/.ci/java-versions.properties#L16.

@peternied @peterzhuamazon where/how do we make this happen?

[peterzhuamazon]

After opensearch-project/OpenSearch#1368, the gradle check is still using Java 14. This comes hardcoded from the infrastructure that runs these. We need it to start picking up Java version from https://github.com/opensearch-project/OpenSearch/blob/main/.ci/java-versions.properties#L16.

@peternied @peterzhuamazon where/how do we make this happen?

Gradle check is running by using the JDK14 defined in Jenkins workflows.
As of now this is a Jenkins only setting as we havent fully migrate freestyle to pipeline and remove that dependency.
One way is to remove it and use the new docker image after we build and push to hub.

[dblock]

I enabled building OpenSearch 2.0 in #850, which should already be using JDK11.

I think we only need the Gradle Checks that runs against OpenSearch main to start using JDK 11 to close this.

OpenSearch has CI defined in a private freestyle project that runs gradle check on OpenSearch#main, branches and PRs. Extract that into Jenkinsfile that is committed to the OpenSearch repo, similar to https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/Jenkinsfile.

[peternied]

@peterzhuamazon Can you into transitioning the existing gradle check job to a jenkinsfile that can reuse the jvm picking logic?

[peterzhuamazon]

@peterzhuamazon Can you into transitioning the existing gradle check job to a jenkinsfile that can reuse the jvm picking logic?

Will put that as a next ticket to work on.
Thanks.

[dblock]

I opened opensearch-project/OpenSearch#2395 for the JDK problem specifically to link it from opensearch-project/opensearch-plugins#64