[CI] o.o.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability

[nknize]

Hit this non-reproducible bug on #2498

./gradlew ':server:internalClusterTest' --tests "org.opensearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability" -Dtests.seed=9A51991686B5895B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-IQ -Dtests.timezone=Pacific/Johnston -Druntime.java=17

org.opensearch.versioning.ConcurrentSeqNoVersioningIT > testSeqNoCASLinearizability FAILED
    MasterNotDiscoveredException[FailedToCommitClusterStateException[publication failed]; nested: OpenSearchException[publication cancelled before committing: timed out after 5s];]; nested: FailedToCommitClusterStateException[publication failed]; nested: OpenSearchException[publication cancelled before committing: timed out after 5s];
        at __randomizedtesting.SeedInfo.seed([9A51991686B5895B:419FB77C9CA7ABEF]:0)
        at org.opensearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:275)
        at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:369)
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:287)
        at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:692)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:833)

[xuezhou25]

Tried running the test for 100 times. It passed successfully.

$ ./gradlew ':server:internalClusterTest' --tests "org.opensearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability" -Dtests.iters=60 

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6
  OS Info               : Linux 5.15.0-1026-aws (amd64)
  JDK Version           : 11 (Eclipse Temurin JDK)
  JAVA_HOME             : /usr/lib/jvm/jdk-11.0.14.1+1
  Random Testing Seed   : 99F9B0821CB45618
  In FIPS 140 mode      : false
=======================================

> Task :server:internalClusterTest
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/home/ftianli/github/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ftianli/.gradle/wrapper/dists/gradle-7.6-all/9f832ih6bniajn45pbmqhk2cw/gradle-7.6/lib/plugins/gradle-testing-base-7.6.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 16m 13s
43 actionable tasks: 1 executed, 42 up-to-date

[reta]

https://build.ci.opensearch.org/job/gradle-check/30329/testReport/junit/org.opensearch.versioning/ConcurrentSeqNoVersioningIT/testSeqNoCASLinearizability/

ClusterManagerNotDiscoveredException[null]
	at app//org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:350)
	at app//org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)
	at app//org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)
	at app//org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:709)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:852)
	at java.base@21.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@21.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@21.0.1/java.lang.Thread.run(Thread.java:1583)

[aasom143]

Ran test for 5000 times, it passed successfully.

[ec2-user@ip-172-31-39-104 OpenSearch]$ ./gradlew ':server:internalClusterTest' --tests "org.opensearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability" -Dtests.seed=9A51991686B5895B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-IQ -Dtests.timezone=Pacific/Johnston -Dtests.timeoutSuite=108000000! -Dtests.iters=5000
Starting a Gradle Daemon (subsequent builds will be faster)
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.7
  OS Info               : Linux 6.1.84-99.169.amzn2023.x86_64 (amd64)
  JDK Version           : 21 (Amazon Corretto JRE)
  JAVA_HOME             : /usr/lib/jvm/java-21-amazon-corretto.x86_64
  Random Testing Seed   : 9A51991686B5895B
  In FIPS 140 mode      : false
=======================================

> Task :server:internalClusterTest
WARNING: Using incubator modules: jdk.incubator.vector
Apr 24, 2024 8:09:30 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/home/ec2-user/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ec2-user/.gradle/wrapper/dists/gradle-8.7-all/aan3ydargesu18aqyqjwhr3pc/gradle-8.7/lib/plugins/gradle-testing-base-8.7.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 19h 23m 34s
57 actionable tasks: 1 executed, 56 up-to-date
WARNING: The following functionality has been deprecated and will be removed in the next major release of the Develocity Gradle plugin. For assistance with migration, see https://gradle.com/help/gradle-plugin-develocity-migration.
- The deprecated "gradle.enterprise.testretry.enabled" system property has been replaced by "develocity.testretry.enabled"
- The "com.gradle.enterprise" plugin has been replaced by "com.gradle.develocity"