[BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes #891

b4sjoo · 2023-05-11T19:36:31Z

What is the bug?
ML-Commons Plugin 3.0 build fails due to jackson update in OS core

How can one reproduce the bug?

./gradlew build

 FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'opensearch-ml'.
> java.util.concurrent.ExecutionException: org.gradle.api.GradleException: Failed to create Jar file /home/runner/.gradle/caches/jars-9/bc33e077f18aab4d8905d3690c9e67e4/jackson-core-2.15.0.jar.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

The text was updated successfully, but these errors were encountered:

b4sjoo · 2023-05-11T19:44:16Z

This is a bug in Gradle - gradle/gradle#24390
Basically we are trying to bump gradle to a version higher than 7.6
gradle migration guide

b4sjoo · 2023-05-11T19:47:22Z

We need to bump gradle to version 8 because ./gradlew wrapper in gradle 7.6 threw following error messages:

* What went wrong:
An exception occurred applying plugin request [id: 'com.github.johnrengelman.shadow']
> Failed to apply plugin class 'com.github.jengelman.gradle.plugins.shadow.ShadowBasePlugin'.
   > This version of Shadow supports Gradle 8.0+ only. Please upgrade.

following fixing PRs opensearch-project/neural-search#169, opensearch-project/k-NN#899

b4sjoo · 2023-05-11T20:03:13Z

After bumping ./gradlew wrapper returned error message like

FAILURE: Build completed with 1 failures.
1: Task failed with an exception.
-----------
* What went wrong:
A problem occurred configuring project ':opensearch-ml-plugin'.
> Failed to notify project evaluation listener.
   > Could not create task ':opensearch-ml-plugin:buildDeb'.
      > You can't map a property that does not exist: propertyName=version
   > Could not create task ':opensearch-ml-plugin:buildRpm'.
      > You can't map a property that does not exist: propertyName=version

buildDeb and buildRpm are two tasks imported by package nebula.ospackage. It also need to be updated so that it can be compatible with gradle 8

b4sjoo · 2023-05-11T22:28:01Z

./gradlew build failed in test step:

> Task :opensearch-ml-algorithms:compileTestJava FAILED
/local/home/seasonsg/dev/ml-commons/ml-algorithms/src/test/java/org/opensearch/ml/engine/algorithms/anomalylocalization/AnomalyLocalizerImplTests.java:40: error: cannot find symbol
import org.opensearch.common.collect.ImmutableOpenMap;
                                    ^
  symbol:   class ImmutableOpenMap
  location: package org.opensearch.common.collect
1 error

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-algorithms:compileTestJava'.
> Compilation failed; see the compiler error output for details.

This is due to OS core breaking changes opensearch-project/OpenSearch#7309

b4sjoo · 2023-05-11T22:28:42Z

Following this PR opensearch-project/k-NN#885 to fix this problem.

b4sjoo · 2023-05-11T23:00:09Z

./gradlew build failed in test step:

> Task :opensearch-ml-algorithms:compileTestJava
Note: /local/home/seasonsg/dev/ml-commons/ml-algorithms/src/test/java/org/opensearch/ml/engine/algorithms/anomalylocalization/AnomalyLocalizerImplTests.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :opensearch-ml-algorithms:test
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsOverallDecrease FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsForSearchFailure FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testExecuteFail FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsFilterEntity FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsGivenIndexPattern FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testExecuteInterrupted FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testExecuteSucceed FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsGivenNoAnomaly FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsForInvalidIndexName FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsOverallUnchange FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsForInvalidTimeRange FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449

org.opensearch.ml.engine.algorithms.anomalylocalization.AnomalyLocalizerImplTests > testGetLocalizedResultsGivenAnomaly FAILED
    java.lang.NullPointerException at AnomalyLocalizerImplTests.java:449
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/djl_cache_2bb316be-484a-439c-a7cb-014ea30f3e25/ml_cache/pytorch/1.12.1-cpu-precxx11-linux-x86_64/libtorch_cpu.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

142 tests completed, 12 failed

> Task :opensearch-ml-algorithms:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-algorithms:test'.

b4sjoo · 2023-05-11T23:04:12Z

This is due to breaking changes in OS core: opensearch-project/OpenSearch#7301, the customs field in ClusterState class was changed to Map<String, Custom> from ImmutableOpenMap<String, Custom>. Following this PR to fix: opensearch-project/anomaly-detection#886

bbarani · 2023-05-12T02:16:43Z

Tagging @nknize @reta @saratvemulapalli

b4sjoo · 2023-05-12T18:24:04Z

The migration from ImmutableOpenMap to java.util.Map finished and can confirm succeed. But now the integTest will fail due to deployModel will fail in RestMLCustomModelActionIT

> Task :opensearch-ml-plugin:integTest


org.opensearch.ml.rest.RestMLCustomModelActionIT > testCustomModelWorkflow FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17


org.opensearch.ml.rest.RestMLCustomModelActionIT > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17

reta · 2023-05-12T19:02:53Z

@bbarani @b4sjoo the plugin has to be updated (sadly)

ylwu-amzn · 2023-05-12T19:43:35Z

Another breaking change is the string length limit from this PR
opensearch-project/OpenSearch#7286 (comment)

By bumping jackson version, we will introduce the string length limitation FasterXML/jackson-core#1001 (comment)

The model content saved as Base64 String in index. As model content string exceeds the string limit, our IT failed. @reta created an opensearch-project/OpenSearch#7549 to support tuning string length limit, before that ready, we have to disable our IT, also neural-search plugin has to disable their IT

b4sjoo · 2023-05-16T00:21:59Z

The migration from ImmutableOpenMap to java.util.Map finished and can confirm succeed. But now the integTest will fail due to deployModel will fail in RestMLCustomModelActionIT

> Task :opensearch-ml-plugin:integTest


org.opensearch.ml.rest.RestMLCustomModelActionIT > testCustomModelWorkflow FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17


org.opensearch.ml.rest.RestMLCustomModelActionIT > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([39C4FE8636E3D27E]:0)
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLCustomModelActionIT.testCustomModelWorkflow" -Dtests.seed=39C4FE8636E3D27E -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Africa/Lubumbashi -Druntime.java=17

As we root causing this issue, we found

The taskid returned in deployModel in RestMLCustomModelActionIT is always a FAILED.
The error message in the response is "error":"{\"Fail to find model chunk {MODEL_ID}_0\"}"

Trying to use this code to continue the RCA.

Update: The PR #884 can reproduce this problem, but definitely not RC because main branch has not merged this.

penghuo · 2023-05-16T15:48:53Z

Could we have quick fix before RCA? There are 5 version bump PR pending on this #850 and SQL main branch bwc IT pending on this

ylwu-amzn · 2023-05-16T16:29:51Z

@b4sjoo can we ignore the failed IT/UT to unblock other teams first?

reta · 2023-05-16T17:07:45Z

@penghuo @ylwu-amzn the fix for Jackson 2.15 content limits went in to main (opensearch-project/OpenSearch#7550)

b4sjoo · 2023-05-16T17:17:26Z

@b4sjoo can we ignore the failed IT/UT to unblock other teams first?

So we simply delete the problematic IT first?

ylwu-amzn · 2023-05-16T17:19:26Z

@b4sjoo can we ignore the failed IT/UT to unblock other teams first?

So we simply delete the problematic IT first?

Refer to

ml-commons/plugin/src/test/java/org/opensearch/ml/rest/RestMLSearchTaskActionIT.java

Line 22 in 194fb59

@Ignore

b4sjoo · 2023-05-17T00:15:46Z

Problematic IT was ignored and build passed.

b4sjoo · 2023-05-17T00:16:30Z

Github CI workflow failed due to

> Task :opensearch-ml-client:publishShadowPublicationToMavenLocal FAILED


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
FAILURE: Build failed with an exception.


You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings
* What went wrong:
16 actionable tasks: 10 executed, 6 up-to-date
A problem was found with the configuration of task ':opensearch-ml-client:publishShadowPublicationToMavenLocal' (type 'PublishToMavenLocal').
  - Gradle detected a problem with the following location: '/home/runner/work/ml-commons/ml-commons/client/build/libs/opensearch-ml-client-3.0.0.0-SNAPSHOT.jar'.
    
    Reason: Task ':opensearch-ml-client:publishShadowPublicationToMavenLocal' uses this output of task ':opensearch-ml-client:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
    
    Possible solutions:
      1. Declare task ':opensearch-ml-client:jar' as an input of ':opensearch-ml-client:publishShadowPublicationToMavenLocal'.
      2. Declare an explicit dependency on ':opensearch-ml-client:jar' from ':opensearch-ml-client:publishShadowPublicationToMavenLocal' using Task#dependsOn.
      3. Declare an explicit dependency on ':opensearch-ml-client:jar' from ':opensearch-ml-client:publishShadowPublicationToMavenLocal' using Task#mustRunAfter.
    
    Please refer to https://docs.gradle.org/8.1.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

b4sjoo · 2023-05-17T00:17:57Z

Add a task configuration tasks.named("jar").configure { dependsOn("publishShadowPublicationToMavenLocal") } can make the PublishToMavenLocal task succeed.

b4sjoo · 2023-05-18T22:05:41Z

Jenkins CI workflow on OS 3.0 failed with message:

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' (type 'PublishToMavenRepository').
  - Gradle detected a problem with the following location: 'C:\Users\Administrator\AppData\Local\Temp\tmpypnhpv6j\ml-commons\client\build\libs\opensearch-ml-client-3.0.0.0.jar'.
    
    Reason: Task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' uses this output of task ':opensearch-ml-client:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.\

https://build.ci.opensearch.org/job/distribution-build-opensearch/7856/console

We try to fix it with PR #901

b4sjoo · 2023-05-18T23:15:16Z

Here is an issue board to track most of the breaking changes in OS core in 3.0, please keep an eye on it - opensearch-project/OpenSearch#5910

b4sjoo · 2023-05-18T23:17:43Z

And also a place to provide feedback - opensearch-project/opensearch-devops#114

b4sjoo · 2023-05-18T23:41:11Z

Which is due to major refactor in OS core 3.0:

dev/ml-commons/plugin/src/main/java/org/opensearch/ml/cluster/DiscoveryNodeHelper.java:40: error: cannot find symbol
        excludedNodeNames = Strings.commaDelimitedListToSet(ML_COMMONS_EXCLUDE_NODE_NAMES.get(settings));
                                   ^
  symbol:   method commaDelimitedListToSet(String)
  location: class Strings
dev/ml-commons/plugin/src/main/java/org/opensearch/ml/cluster/DiscoveryNodeHelper.java:43: error: cannot find symbol
            .addSettingsUpdateConsumer(ML_COMMONS_EXCLUDE_NODE_NAMES, it -> excludedNodeNames = Strings.commaDelimitedListToSet(it));
                                                                                                       ^
  symbol:   method commaDelimitedListToSet(String)
  location: class Strings

opensearch-project/OpenSearch#7508
However after change the import we got:

dev/ml-commons/common/src/test/java/org/opensearch/ml/common/dataframe/DoubleValueTest.java:54: error: method toString in class Object cannot be applied to given types;
        String jsonStr = Strings.toString(builder);
                                ^
  required: no arguments
  found:    XContentBuilder
  reason: actual and formal argument lists differ in length

Here is a temporary fix method:
opensearch-project/security#2782

b4sjoo · 2023-05-18T23:44:13Z

Created a fix PR #901

I got following integ test error message after migration from org.opensearch.common.Strings to org.opensearch.core.common.Strings

»  java.lang.NoClassDefFoundError: org/opensearch/core/common/Strings
»  	at org.opensearch.ml.cluster.DiscoveryNodeHelper.<init>(DiscoveryNodeHelper.java:40)
»  	at org.opensearch.ml.plugin.MachineLearningPlugin.createComponents(MachineLearningPlugin.java:233)
»  	at org.opensearch.node.Node.lambda$new$16(Node.java:770)
»  	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
»  	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
»  	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
»  	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
»  	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
»  	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
»  	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
»  	at org.opensearch.node.Node.<init>(Node.java:784)
»  	at org.opensearch.node.Node.<init>(Node.java:377)
»  	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
»  	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
»  	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
»  	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
»  	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
»  	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
»  	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
»  	at org.opensearch.cli.Command.main(Command.java:101)
»  	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
»  	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

The line for DiscoveryNodeHelper.java:40 is

excludedNodeNames = Strings.commaDelimitedListToSet(ML_COMMONS_EXCLUDE_NODE_NAMES.get(settings));

And UnitTests were passed. Any thoughts?

b4sjoo · 2023-05-31T23:22:26Z

The integ test error is due to docker image not update. Now integ tests were passed. Issue resolved.

b4sjoo · 2023-06-08T02:04:55Z

Jenkins CI workflow on OS 3.0 failed with message:

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' (type 'PublishToMavenRepository').
  - Gradle detected a problem with the following location: 'C:\Users\Administrator\AppData\Local\Temp\tmpypnhpv6j\ml-commons\client\build\libs\opensearch-ml-client-3.0.0.0.jar'.
    
    Reason: Task ':opensearch-ml-client:publishShadowPublicationToStagingRepository' uses this output of task ':opensearch-ml-client:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.\

https://build.ci.opensearch.org/job/distribution-build-opensearch/7856/console

We try to fix it with PR #901

We did another fix in #962 due to ml-commons not building when publishing ml-client artifacts.

b4sjoo · 2023-06-09T18:44:32Z

After opensearch-build team migrate from jdk17 to jdk20, task :opensearch-ml-plugin:delombok failed with following message:

Exception in thread "main" java.lang.NoSuchFieldError: var
	at lombok.delombok.PrettyPrinter.visitForeachLoop(PrettyPrinter.java:1256)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCEnhancedForLoop.accept(JCTree.java:1232)
	at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
	at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:240)
	at lombok.delombok.PrettyPrinter.visitBlock(PrettyPrinter.java:1133)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCBlock.accept(JCTree.java:1088)
	at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
	at lombok.delombok.PrettyPrinter.visitMethodDef(PrettyPrinter.java:857)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:912)
	at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
	at lombok.delombok.PrettyPrinter.printClassMembers(PrettyPrinter.java:617)
	at lombok.delombok.PrettyPrinter.visitClassDef(PrettyPrinter.java:556)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:810)
	at lombok.delombok.PrettyPrinter.print(PrettyPrinter.java:226)
	at lombok.delombok.PrettyPrinter.visitTopLevel(PrettyPrinter.java:482)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:549)
	at lombok.delombok.DelombokResult.print(DelombokResult.java:74)
	at lombok.delombok.Delombok.delombok(Delombok.java:819)
	at lombok.delombok.Delombok.main(Delombok.java:326)
	at lombok.delombok.DelombokApp.runDirectly(DelombokApp.java:176)
	at lombok.delombok.DelombokApp.runApp(DelombokApp.java:47)
	at lombok.core.Main.go(Main.java:130)
	at lombok.core.Main.main(Main.java:43)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at lombok.launch.Main.main(Main.java:46)

> Task :opensearch-ml-plugin:delombok FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':opensearch-ml-plugin:delombok'.
> Process 'command '/opt/java/openjdk-20/bin/java'' finished with non-zero exit value 1

dhrubo-os · 2023-06-09T18:57:48Z

I think Exception in thread "main" java.lang.NoSuchFieldError: var issue can be solved by allowing here

But when I try to build ml-commons package with Java 20, 12 tests are failing:

java.lang.instrument.IllegalClassFormatException: Error while instrumenting sun/util/resources/cldr/provider/CLDRLocaleDataMetaInfo with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:94)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
        at java.base/java.lang.ClassLoader.defineClass2(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1104)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:182)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(BuiltinClassLoader.java:741)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClass(BuiltinClassLoader.java:621)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:633)
        at java.base/java.lang.Class.forName(Class.java:585)
        at java.base/java.lang.Class.forName(Class.java:560)
        at java.base/java.util.ServiceLoader.loadProvider(ServiceLoader.java:859)
        at java.base/java.util.ServiceLoader$ModuleServicesLookupIterator.hasNext(ServiceLoader.java:1084)
        at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
        at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
        at java.base/sun.util.cldr.CLDRLocaleProviderAdapter.lambda$new$0(CLDRLocaleProviderAdapter.java:84)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
        at java.base/sun.util.cldr.CLDRLocaleProviderAdapter.<init>(CLDRLocaleProviderAdapter.java:83)
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.forType(LocaleProviderAdapter.java:189)
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.findAdapter(LocaleProviderAdapter.java:288)
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.getAdapter(LocaleProviderAdapter.java:259)
        at java.base/java.util.Calendar.createCalendar(Calendar.java:1687)
        at java.base/java.util.Calendar.getInstance(Calendar.java:1655)
        at java.base/java.text.SimpleDateFormat.initializeCalendar(SimpleDateFormat.java:680)
        at java.base/java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:624)
        at java.base/java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.getLastResortErrorLogFile(SystemApplicationClassLoaderWorker.java:129)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:97)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.io.IOException: Error while instrumenting sun/util/resources/cldr/provider/CLDRLocaleDataMetaInfo with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrumentError(Instrumenter.java:161)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:111)
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:92)
        ... 35 more
Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 64
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:199)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:180)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:166)
        at org.jacoco.agent.rt.internal_b6258fc.core.internal.instr.InstrSupport.classReaderFor(InstrSupport.java:280)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:77)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:109)
        ... 36 more

Gradle Test Executor 1 started executing tests.

> Task :opensearch-ml-algorithms:test

org.opensearch.ml.engine.MLEngineClassLoaderTests > initInstance_LocalSampleCalculator_RegisterFirst STANDARD_ERROR
    java.lang.instrument.IllegalClassFormatException: Error while instrumenting com/sun/tools/attach/VirtualMachine with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:94)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
        at java.base/java.lang.ClassLoader.defineClass2(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1104)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:182)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:821)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(BuiltinClassLoader.java:741)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:665)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:496)
        at java.base/java.lang.Class.forName(Class.java:475)
        at net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Accessor$Simple.of(ByteBuddyAgent.java:972)
        at net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$ForModularizedVm.attempt(ByteBuddyAgent.java:1096)
        at net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound.attempt(ByteBuddyAgent.java:1276)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:626)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:612)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:564)
        at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:541)
        at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.<clinit>(InlineDelegateByteBuddyMockMaker.java:115)
        at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.<init>(InlineByteBuddyMockMaker.java:23)
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
        at org.mockito.internal.configuration.plugins.PluginInitializer.loadImpl(PluginInitializer.java:53)
        at org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:69)
        at org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:54)
        at org.mockito.internal.configuration.plugins.PluginRegistry.<init>(PluginRegistry.java:28)
        at org.mockito.internal.configuration.plugins.Plugins.<clinit>(Plugins.java:22)
        at org.mockito.internal.MockitoCore.<clinit>(MockitoCore.java:77)
        at org.mockito.Mockito.<clinit>(Mockito.java:1614)
        at org.opensearch.ml.engine.MLEngineClassLoaderTests.initInstance_LocalSampleCalculator_RegisterFirst(MLEngineClassLoaderTests.java:62)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
        at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
        at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at jdk.proxy2/jdk.proxy2.$Proxy5.processTestClass(Unknown Source)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
        at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
    Caused by: java.io.IOException: Error while instrumenting com/sun/tools/attach/VirtualMachine with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrumentError(Instrumenter.java:161)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:111)
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:92)
        ... 74 more
    Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 64
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:199)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:180)
        at org.jacoco.agent.rt.internal_b6258fc.asm.ClassReader.<init>(ClassReader.java:166)
        at org.jacoco.agent.rt.internal_b6258fc.core.internal.instr.InstrSupport.classReaderFor(InstrSupport.java:280)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:77)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:109)
        ... 75 more
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

org.opensearch.ml.engine.MLEngineClassLoaderTests > initInstance_LocalSampleCalculator_RegisterFirst FAILED
    org.mockito.exceptions.base.MockitoException: 
    Mockito cannot mock this class: interface org.opensearch.client.Client.

    If you're not sure why you're getting this error, please report to the mailing list.


    Java               : 20
    JVM vendor name    : Amazon.com Inc.
    JVM vendor version : 20.0.1+9-FR
    JVM name           : OpenJDK 64-Bit Server VM
    JVM version        : 20.0.1+9-FR
    JVM info           : mixed mode, sharing
    OS name            : Mac OS X
    OS version         : 13.4


    You are seeing this disclaimer because Mockito is configured to create inlined mocks.
    You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc.

    Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [interface org.opensearch.client.OpenSearchClient, interface org.opensearch.client.Client, interface java.io.Closeable, interface java.lang.AutoCloseable, interface org.opensearch.core.common.lease.Releasable]
        at app//org.opensearch.ml.engine.MLEngineClassLoaderTests.initInstance_LocalSampleCalculator_RegisterFirst(MLEngineClassLoaderTests.java:62)

        Caused by:
        org.mockito.exceptions.base.MockitoException: Could not modify all classes [interface org.opensearch.client.OpenSearchClient, interface org.opensearch.client.Client, interface java.io.Closeable, interface java.lang.AutoCloseable, interface org.opensearch.core.common.lease.Releasable]
            at app//net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:157)
            at app//net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:371)
            at app//net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:179)
            at app//net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:382)
            ... 1 more

            Caused by:
            java.lang.IllegalStateException: 
            Byte Buddy could not instrument all classes within the mock's type hierarchy

            This problem should never occur for javac-compiled classes. This problem has been observed for classes that are:
             - Compiled by older versions of scalac
             - Classes that are part of the Android distribution
                at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:280)
                at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.mockClass(InlineBytecodeGenerator.java:213)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.lambda$mockClass$0(TypeCachingBytecodeGenerator.java:47)
                at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:157)
                at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:371)
                at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:179)
                at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:382)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:40)
                at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.createMockType(InlineDelegateByteBuddyMockMaker.java:389)
                at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.doCreateMock(InlineDelegateByteBuddyMockMaker.java:349)
                at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.createMock(InlineDelegateByteBuddyMockMaker.java:328)
                at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.createMock(InlineByteBuddyMockMaker.java:56)
                at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:53)
                at org.mockito.internal.MockitoCore.mock(MockitoCore.java:96)
                at org.mockito.Mockito.mock(Mockito.java:1965)
                at org.mockito.Mockito.mock(Mockito.java:1880)
                ... 1 more

                Caused by:
                java.lang.IllegalArgumentException: Java 20 (64) is not supported by the current version of Byte Buddy which officially supports Java 18 (62) - update Byte Buddy or set net.bytebuddy.experimental as a VM property
                    at net.bytebuddy.utility.OpenedClassReader.of(OpenedClassReader.java:96)
                    at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:3948)
                    at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2213)
                    at net.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:224)
                    at net.bytebuddy.dynamic.scaffold.inline.AbstractInliningDynamicTypeBuilder.make(AbstractInliningDynamicTypeBuilder.java:123)
                    at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3661)
                    at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.transform(InlineBytecodeGenerator.java:394)
                    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:244)
                    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
                    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:541)
                    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
                    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:169)
                    at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:276)
                    ... 16 more

Mainly the issue i think:

Caused by:
                java.lang.IllegalArgumentException: Java 20 (64) is not supported by the current version of Byte Buddy which officially supports Java 18 (62) - update Byte Buddy or set net.bytebuddy.experimental as a VM property

b4sjoo · 2023-06-09T19:02:07Z

Yeah it seems there will be many breaking changes after migrating to jdk20, sigh...

dhrubo-os · 2023-06-10T00:00:50Z

Update:

We can reproduce the same error in our end with executing: ./gradlew :opensearch-ml-plugin:delombok


The :opensearch-ml-plugin:delombok task failed.View task in console log
Process 'command '/Library/Java/JavaVirtualMachines/amazon-corretto-20.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1
Exception
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':opensearch-ml-plugin:delombok'.	
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:149)	
	at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)	
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:147)	
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)	
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)	
	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)	
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)	
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)	
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)	
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)	
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)	
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)	
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)	
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)	
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)	
Caused by: org.gradle.process.internal.ExecException: Process 'command '/Library/Java/JavaVirtualMachines/amazon-corretto-20.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1	
	at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:415)	
	at org.gradle.process.internal.DefaultJavaExecAction.execute(DefaultJavaExecAction.java:52)	
	at org.gradle.process.internal.DefaultExecActionFactory.javaexec(DefaultExecActionFactory.java:195)	
	at org.gradle.process.internal.DefaultExecOperations.javaexec(DefaultExecOperations.java:42)	
	at io.freefair.gradle.plugins.lombok.tasks.Delombok.delombok(Delombok.java:198)	
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)	
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)	
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)	
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)	
	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)	
	at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:242)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)	
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)	
	at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:227)	
	at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:210)	
	at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:193)	
	at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)	
	at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)	
	at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)	
	at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)	
	at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)	
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)	
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56)	
	at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)	
	at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)	
	at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)	
	at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)	
	at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)	
	at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)	
	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)	
	at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28)	
	at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)	
	at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)	
	at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)	
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40)	
	at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29)	
	at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:166)	
	at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)	
	at org.gradle.internal.Either$Right.fold(Either.java:175)	
	at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)	
	at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)	
	at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)	
	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)	
	at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)	
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)	
	at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)	
	at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)	
	at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)	
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)	
	at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)	
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)	
	at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)	
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)	
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)	
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:76)	
	at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:37)	
	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:94)	
	at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:49)	
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71)	
	at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45)	
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177)	
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:86)	
	at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53)	
	at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)	
	at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)	
	at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)	
	at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)	
	at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)	
	at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75)	
	at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41)	
	at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)	
	at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:287)	
	at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)	
	at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21)	
	at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)	
	at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)	
	at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)	
	at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)	
	at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)	
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146)	
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)	
	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)	
	at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)	
	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)	
	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)	
	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)	
	at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)	
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)	
	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)	
	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)	
	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)	
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)	
	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)	
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)	
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)

reta · 2023-06-10T12:15:20Z

Yeah it seems there will be many breaking changes after migrating to jdk20, sigh...

Do you folks need help with that? The lombok is real bummer for any recent JDK updates, Mockito 5.x has no issues with JDK-20

dhrubo-os · 2023-06-12T18:29:17Z

@reta Right now OS is building failing due to delombok issue, we created an issue in freefair.

If you have any suggestion about the delombok issue that will be helpful.

In addition, we are facing build error:

Caused by: java.io.IOException: Error while instrumenting sun/util/resources/cldr/provider/CLDRLocaleDataMetaInfo with JaCoCo 0.8.8.202204050719/5dcf34a.
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrumentError(Instrumenter.java:161)
        at org.jacoco.agent.rt.internal_b6258fc.core.instr.Instrumenter.instrument(Instrumenter.java:111)
        at org.jacoco.agent.rt.internal_b6258fc.CoverageTransformer.transform(CoverageTransformer.java:92)
        ... 35 more
Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 64

We are adding jacoco dependencies here.

If you have any suggestion how to solve this issue, that will also be helpful.

Thanks.

reta · 2023-06-15T14:25:12Z

@dhrubo-os anything left here you need help with folks? or we could close it :)

dhrubo-os · 2023-06-15T15:01:00Z

@reta I think that's all the support we needed for now. Thanks a lot. I'm waiting for a successful build for OS build issue. After confirming this, I'll close the issue.

dhrubo-os · 2023-06-15T15:21:47Z

Tested the build in my end and the build was successful. Closing the issue. Thanks.

b4sjoo · 2023-08-07T22:59:56Z

Here is a temporary fix method:
opensearch-project/security#2782

#1191 Provide a final solution to string methods

b4sjoo · 2023-08-11T20:30:17Z

More refactoring from OS core - opensearch-project/OpenSearch#8157
Fixed in PR #1122

b4sjoo added bug Something isn't working untriaged labels May 11, 2023

b4sjoo changed the title ~~[BUG] ML Plugin build failure due to~~ [BUG] ML Plugin build failure due to OpenSearch Core breaking changes May 11, 2023

b4sjoo changed the title ~~[BUG] ML Plugin build failure due to OpenSearch Core breaking changes~~ [BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes May 11, 2023

b4sjoo self-assigned this May 11, 2023

b4sjoo mentioned this issue May 11, 2023

SQL Plugin bump version failed #850

Closed

b4sjoo mentioned this issue May 11, 2023

Bumping gradle from 7.4 to 8.1 to unblock build pipeline #892

Merged

5 tasks

gaiksaya mentioned this issue May 12, 2023

Remove ml-commons and neural search from 3.0.0 opensearch-project/opensearch-build#3511

Merged

ylwu-amzn removed the untriaged label May 15, 2023

ylwu-amzn mentioned this issue May 16, 2023

[AUTOCUT] OS Distribution Build Failed for sql-3.0.0 opensearch-project/sql#1630

Closed

b4sjoo mentioned this issue May 18, 2023

fix Jenkins CI pipeline for OS 3.0 version bump #901

Merged

5 tasks

b4sjoo reopened this May 18, 2023

github-actions bot added the untriaged label May 18, 2023

b4sjoo removed the untriaged label May 18, 2023

peterzhuamazon mentioned this issue May 25, 2023

[RELEASE] Release version 2.8.0 opensearch-project/opensearch-build#3434

Closed

42 tasks

b4sjoo closed this as completed May 31, 2023

b4sjoo mentioned this issue Jun 9, 2023

[AUTOCUT] OS Distribution Build Failed for ml-commons-3.0.0 #965

Closed

b4sjoo reopened this Jun 9, 2023

github-actions bot added the untriaged label Jun 9, 2023

dhrubo-os removed the untriaged label Jun 12, 2023

reta mentioned this issue Jun 14, 2023

Add JDK-20 support #986

Merged

5 tasks

dhrubo-os closed this as completed Jun 15, 2023

[BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes #891

[BUG] ML-Commons Plugin build failure due to OpenSearch Core breaking changes #891

Comments

b4sjoo commented May 11, 2023 • edited Loading

b4sjoo commented May 11, 2023 • edited Loading

b4sjoo commented May 11, 2023 • edited Loading

b4sjoo commented May 11, 2023 • edited Loading

b4sjoo commented May 11, 2023

b4sjoo commented May 11, 2023 • edited Loading

b4sjoo commented May 11, 2023

b4sjoo commented May 11, 2023 • edited Loading

bbarani commented May 12, 2023

b4sjoo commented May 12, 2023 • edited Loading

reta commented May 12, 2023

ylwu-amzn commented May 12, 2023

b4sjoo commented May 16, 2023 • edited Loading

penghuo commented May 16, 2023 • edited Loading

ylwu-amzn commented May 16, 2023

reta commented May 16, 2023

b4sjoo commented May 16, 2023

ylwu-amzn commented May 16, 2023

b4sjoo commented May 17, 2023

b4sjoo commented May 17, 2023

b4sjoo commented May 17, 2023

b4sjoo commented May 18, 2023

b4sjoo commented May 18, 2023

b4sjoo commented May 18, 2023

b4sjoo commented May 18, 2023

b4sjoo commented May 18, 2023 • edited Loading

b4sjoo commented May 31, 2023

b4sjoo commented Jun 8, 2023

b4sjoo commented Jun 9, 2023

dhrubo-os commented Jun 9, 2023 • edited Loading

b4sjoo commented Jun 9, 2023

dhrubo-os commented Jun 10, 2023

reta commented Jun 10, 2023

dhrubo-os commented Jun 12, 2023

reta commented Jun 15, 2023

dhrubo-os commented Jun 15, 2023

dhrubo-os commented Jun 15, 2023

b4sjoo commented Aug 7, 2023

b4sjoo commented Aug 11, 2023

b4sjoo commented May 11, 2023 •

edited

Loading

b4sjoo commented May 11, 2023 •

edited

Loading

b4sjoo commented May 11, 2023 •

edited

Loading

b4sjoo commented May 11, 2023 •

edited

Loading

b4sjoo commented May 11, 2023 •

edited

Loading

b4sjoo commented May 11, 2023 •

edited

Loading

b4sjoo commented May 12, 2023 •

edited

Loading

b4sjoo commented May 16, 2023 •

edited

Loading

penghuo commented May 16, 2023 •

edited

Loading

b4sjoo commented May 18, 2023 •

edited

Loading

dhrubo-os commented Jun 9, 2023 •

edited

Loading