Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Intermittent java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). #1843

Closed
dreamer-89 opened this issue Jan 4, 2022 · 12 comments
Closed

[BUG] Intermittent java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). #1843

dreamer-89 opened this issue Jan 4, 2022 · 12 comments
Assignees
@dreamer-89
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run

Comments

@dreamer-89
Copy link
Member

Describe the bug
Intermittent test failures due to java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). Sample jstack during timeout

  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@528534bb
  2> 	at java.base@17.0.1/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@528534bb
  2> 	at java.base@17.0.1/java.lang.Thread.join(Thread.java:1304)
  2> 	at java.base@17.0.1/java.lang.Thread.join(Thread.java:1372)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
  2> 	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  2> 	at java.base@17.0.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@17.0.1/java.lang.reflect.Method.invoke(Method.java:568)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at jdk.proxy2/jdk.proxy2.$Proxy5.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@17.0.1/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@17.0.1/java.lang.ref.Reference.processPendingReferences(Reference.java:253)
  2> 	at java.base@17.0.1/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:215)

  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@22e1ea1d
  2> 	at java.base@17.0.1/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@22e1ea1d
  2> 	at java.base@17.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@17.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@17.0.1/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:172)

  2> "Signal Dispatcher" ID=4 RUNNABLE

  2> "Common-Cleaner" ID=23 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@2109b750
  2> 	at java.base@17.0.1/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@2109b750
  2> 	at java.base@17.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@17.0.1/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	at java.base@17.0.1/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:162)

  2> "Notification Thread" ID=25 RUNNABLE

  2> "/127.0.0.1:39152 to /127.0.0.1:43503 workers" ID=27 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56d2f773
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56d2f773
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  2> 	at java.base@17.0.1/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:370)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@7397c1cf
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2> 	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  2> 	at java.base@17.0.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@17.0.1/java.lang.reflect.Method.invoke(Method.java:568)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@24c22fe

  2> "/127.0.0.1:39152 to /127.0.0.1:43503 workers Thread 2" ID=28 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a0d79ce
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a0d79ce
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java  2> :49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@5170bcf4

  2> "/127.0.0.1:39152 to /127.0.0.1:43503 workers Thread 3" ID=29 RUNNABLE (in native code)
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@77d95ef9
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7e46b5e4
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:146)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:82)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@f99f5e0

  2> "SUITE-ShardIndexingPressureSettingsIT-seed#[CC90EB4C1A127257]" ID=1230 RUNNABLE
  2> 	at java.management@17.0.1/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@17.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@17.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
  2> 	- locked java.lang.Object@3feeac30
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629)

  2> "TEST-ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting-seed#[CC90EB4C1A127257]" ID=1231 WAITING on org.opensearch.common.util.concurrent.BaseFuture$Sync@6e7c610a
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.BaseFuture$Sync@6e7c610a
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
  2> 	at app//org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:267)
  2> 	at app//org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:99)
  2> 	at app//org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:69)
  2> 	at app//org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:50)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT.lambda$testShardIndexingPressureEnforcedEnabledDisabledSetting$2(ShardIndexingPressureSettingsIT.java:407)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT$$Lambda$4279/0x0000000801586f88.run(Unknown Source)
  2> 	at app//org.apache.lucene.util.LuceneTestCase._expectThrows(LuceneTestCase.java:2879)
  2> 	at app//org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2754)
  2> 	at app//org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2749)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting(ShardIndexingPressureSettingsIT.java:407)
  2> 	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  2> 	at java.base@17.0.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@17.0.1/java.lang.reflect.Method.invoke(Method.java:568)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  2> 	at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at app//org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  2> 	at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  2> 	at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evalua  2> te(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at app//org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][[timer]]" ID=1594 TIMED_WAITING
  2> 	at java.base@17.0.1/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:631)

  2> "opensearch[node_t0][scheduler][T#1]" ID=1595 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53efd4aa
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53efd4aa
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
  2> 	at java.base@17.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@17.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][[timer]]" ID=1596 TIMED_WAITING
  2> 	at java.base@17.0.1/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:631)

  2> "opensearch[node_t1][scheduler][T#1]" ID=1597 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18cdb2a6
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18cdb2a6
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
  2> 	at java.base@17.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@17.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t2][[timer]]" ID=1598 TIMED_WAITING
  2> 	at java.base@17.0.1/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.opensearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:631)

  2> "opensearch[node_t2][scheduler][T#1]" ID=1599 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@520812a3
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@520812a3
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
  2> 	at java.base@17.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@17.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][transport_worker][T#1]" ID=1603 RUNNABLE (in native code)
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@5ff15dd9
  2> 	- locked sun.nio.ch.EPollSelectorImpl@72168809
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181)
  2> 	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
  2> 	at app//org.opensearch.nio.NioSelectorGroup$$Lambda$2230/0x0000000801247090.run(Unknown Source)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@73b5c6f

  2> "opensearch[node_t1][transport_worker][T#2]" ID=1604 RUNNABLE
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@15b4cb21
  2> 	- locked sun.nio.ch.EPollSelectorImpl@40fe43db
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181)
  2> 	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
  2> 	at app//org.opensearch.nio.NioSelectorGroup$$Lambda$2230/0x0000000801247090.run(Unknown Source)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@66edd0d

  2> "opensearch[node_t2][transport_worker][T#1]" ID=1605 RUNNABLE
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@5b6b318b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@68351404
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181)
  2> 	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
  2> 	at app//org.opensearch.nio.NioSelectorGroup$$Lambda$2230/0x0000000801247090.run(Unknown Source)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5a677242

  2> "opensearch[node_t0][transport_worker][T#1]" ID=1606 RUNNABLE
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@2a114ab6
  2> 	- locked sun.nio.ch.EPollSelectorImpl@28f4d116
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181)
  2> 	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
  2> 	at app//org.opensearch.nio.NioSelectorGro  2> up$$Lambda$2230/0x0000000801247090.run(Unknown Source)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@68216538

  2> "opensearch[node_t2][transport_worker][T#2]" ID=1607 RUNNABLE
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@9b5ef1c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@29d14066
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181)
  2> 	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
  2> 	at app//org.opensearch.nio.NioSelectorGroup$$Lambda$2230/0x0000000801247090.run(Unknown Source)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@27a3482b

  2> "opensearch[node_t0][transport_worker][T#2]" ID=1608 RUNNABLE
  2> 	at java.base@17.0.1/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@17.0.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
  2> 	- locked sun.nio.ch.Util$2@4448f142
  2> 	- locked sun.nio.ch.EPollSelectorImpl@62a33bd8
  2> 	at java.base@17.0.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:181)
  2> 	at app//org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
  2> 	at app//org.opensearch.nio.NioSelectorGroup$$Lambda$2230/0x0000000801247090.run(Unknown Source)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4152959a

  2> "opensearch[node_t1][generic][T#1]" ID=1609 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][clusterApplierService#updateTask][T#1]" ID=1610 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7fcc53a9
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7fcc53a9
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  2> 	at java.base@17.0.1/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][generic][T#2]" ID=1613 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t2][generic][T#1]" ID=1614 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t2][clusterApplierService#updateTask][T#1]" ID=1615 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51e16135
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51e16135
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  2> 	at java.base@17.0.1/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:53  2> 5)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][generic][T#3]" ID=1618 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t2][generic][T#2]" ID=1619 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t2][generic][T#3]" ID=1620 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t2][generic][T#4]" ID=1621 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4caca73
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][generic][T#4]" ID=1622 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7c0d6f8b
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][generic][T#1]" ID=1623 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1  2> /java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][clusterApplierService#updateTask][T#1]" ID=1624 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bcb7fc7
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5bcb7fc7
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  2> 	at java.base@17.0.1/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][generic][T#2]" ID=1627 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][generic][T#3]" ID=1628 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][generic][T#4]" ID=1629 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@46072db5
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][masterService#updateTask][T#1]" ID=1630 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@980bf9f
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@980bf9f
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
  2> 	at java.base@17.0.1/java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][management][T#1]" ID=1631 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@53328f53
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@53328f53
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:676)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1294)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][management][T#2]" ID=1632 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@53328f53
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Na  2> tive Method)
  2> 	- timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@53328f53
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:676)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1294)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][management][T#1]" ID=1633 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2ca77a05
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2ca77a05
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:676)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1294)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][DanglingIndices#updateTask][T#1]" ID=1634 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2c920d11
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2c920d11
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][DanglingIndices#updateTask][T#1]" ID=1635 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2062335b
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2062335b
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][write][T#1]" ID=1636 WAITING on java.util.concurrent.CountDownLatch$Sync@676d7813
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@676d7813
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
  2> 	at java.base@17.0.1/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT.lambda$blockPrimary$10(ShardIndexingPressureSettingsIT.java:883)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT$$Lambda$4271/0x0000000801585690.sendRequest(Unknown Source)
  2> 	at app//org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:250)
  2> 	at app//org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:869)
  2> 	at app//org.opensearch.transport.TransportService$$Lambda$1835/0x000000080118ad38.sendRequest(Unknown Source)
  2> 	at app//org.opensearch.transport.TransportService.sendRequest(TransportService.java:756)
  2> 	at app//org.opensearch.transport.TransportService.sendRequest(TransportService.java:700)
  2> 	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1311)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:306)
  2> 	at app//org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:123)
  2> 	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
  2> 	at app//org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:296)
  2> 	at app//org.opensearch.action.support.RetryableAction.run(RetryableAction.java:101)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:325)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:226)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:174)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation$$Lambda$3349/0x00000008014071e8.accept(Unknown Source)
  2> 	at app//org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78)
  2> 	at app//org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:178)
  2> 	at app//org.opensearch.action.ActionListener.completeWith(ActionListener.java:351)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:264)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:227)
  2> 	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:276)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:165)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:98)
  2> 	at app//org.ope  2> nsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:220)
  2> 	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792)
  2> 	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@45411f1b

  2> "opensearch[node_t1][write][T#2]" ID=1637 WAITING on java.util.concurrent.CountDownLatch$Sync@676d7813
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@676d7813
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
  2> 	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
  2> 	at java.base@17.0.1/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT.lambda$blockPrimary$10(ShardIndexingPressureSettingsIT.java:883)
  2> 	at app//org.opensearch.index.ShardIndexingPressureSettingsIT$$Lambda$4271/0x0000000801585690.sendRequest(Unknown Source)
  2> 	at app//org.opensearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:250)
  2> 	at app//org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:869)
  2> 	at app//org.opensearch.transport.TransportService$$Lambda$1835/0x000000080118ad38.sendRequest(Unknown Source)
  2> 	at app//org.opensearch.transport.TransportService.sendRequest(TransportService.java:756)
  2> 	at app//org.opensearch.transport.TransportService.sendRequest(TransportService.java:700)
  2> 	at app//org.opensearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1311)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:306)
  2> 	at app//org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:123)
  2> 	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
  2> 	at app//org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:296)
  2> 	at app//org.opensearch.action.support.RetryableAction.run(RetryableAction.java:101)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:325)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:226)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:174)
  2> 	at app//org.opensearch.action.support.replication.ReplicationOperation$$Lambda$3349/0x00000008014071e8.accept(Unknown Source)
  2> 	at app//org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78)
  2> 	at app//org.opensearch.action.ActionListener$4.onResponse(ActionListener.java:178)
  2> 	at app//org.opensearch.action.ActionListener.completeWith(ActionListener.java:351)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:264)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:227)
  2> 	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:276)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:165)
  2> 	at app//org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:98)
  2> 	at app//org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:220)
  2> 	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792)
  2> 	at app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@65fa9bbc

  2> "opensearch[node_t0][write][T#1]" ID=1638 WAITING on java.util.concurrent.LinkedTransferQueue@700d41a9
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@700d41a9
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][write][T#2]" ID=1639 WAITING on java.util.concurrent.LinkedTransferQueue@700d41a9
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@700d41a9
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][write][T#3]" ID=1640 WAITING on java.util.concurrent.LinkedTransferQueue@700d41a9
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@700d41a9
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0  2> .1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][refresh][T#1]" ID=1641 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4c4b203c
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@4c4b203c
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][refresh][T#1]" ID=1642 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7d023a05
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@7d023a05
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][management][T#3]" ID=1645 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@53328f53
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@53328f53
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:676)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1294)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][management][T#2]" ID=1646 TIMED_WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2ca77a05
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@2ca77a05
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:676)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1294)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t0][flush][T#1]" ID=1647 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@71236856
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@71236856
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

  2> "opensearch[node_t1][flush][T#1]" ID=1648 WAITING on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@6bb96fd2
  2> 	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.opensearch.common.util.concurrent.OpenSearchExecutors$ExecutorScalingQueue@6bb96fd2
  2> 	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
  2> 	at java.base@17.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
  2> 	at java.base@17.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)
  2> 	at java.base@17.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  2> 	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

To Reproduce
Issue is not straight forward to reproduce. Need to run the gradle check in a loop multiple times.

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
No test suite timeouts.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@dreamer-89 dreamer-89 added bug Something isn't working untriaged labels Jan 4, 2022
@dreamer-89 dreamer-89 changed the title [BUG] Intermittent java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). #1826 [BUG] Intermittent java.lang.Exception: Suite timeout exceeded (>= 1200000 msec). Jan 4, 2022
@anasalkouz anasalkouz added flaky-test Random test failure that succeeds on second run and removed untriaged labels Jan 4, 2022
@reta
Copy link
Collaborator

reta commented Jan 5, 2022

Tracked here #1715 as well, related to org.opensearch.index.ShardIndexingPressureSettingsIT

@dreamer-89
Copy link
Member Author

dreamer-89 commented Jan 10, 2022 •
edited
Loading

Ran test in batch of 100s but did not get SUITE Timeout error; though got 3 different failures. First failure is Assertion failure on cluster having green state while last two are related with #1767

Command Used

for num in {6100..6199}; do ./gradlew internalClusterTest >> /tmp/gradle_$num.txt 2>&1;  cp server/build/reports/tests/internalClusterTest/classes/org.opensearch.index.ShardIndexingPressureSettingsIT.html /tmp/output_$num;   sleep 10; done;

Commit: 1bb8864
Ec2 host: c5.18xlarge Ubuntu

Gradle failure output

ubuntu@ip-172-31-33-23:~$ grep -A 20 "REPRODUCE WITH"  /tmp/gradle_*
/tmp/gradle_48.txt:REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.flush.FlushIT.testSyncedFlushWithConcurrentIndexing" -Dtests.seed=7E95846E64AC32C1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-LB -Dtests.timezone=America/Mazatlan -Druntime.java=17
/tmp/gradle_48.txt-
/tmp/gradle_48.txt-org.opensearch.indices.flush.FlushIT > testSyncedFlushWithConcurrentIndexing FAILED
/tmp/gradle_48.txt-    java.lang.AssertionError: timed out waiting for green state
/tmp/gradle_48.txt-        at __randomizedtesting.SeedInfo.seed([7E95846E64AC32C1:F36203ACC15FDF3B]:0)
/tmp/gradle_48.txt-        at org.junit.Assert.fail(Assert.java:89)
/tmp/gradle_48.txt-        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:987)
/tmp/gradle_48.txt-        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:926)
/tmp/gradle_48.txt-        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:915)
/tmp/gradle_48.txt-        at org.opensearch.indices.flush.FlushIT.testSyncedFlushWithConcurrentIndexing(FlushIT.java:281)
/tmp/gradle_48.txt-WARNING: A terminally deprecated method in java.lang.System has been called
/tmp/gradle_48.txt-WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.3.3-all/4295vidhdd9hd3gbjyw1xqxpo/gradle-7.3.3/lib/plugins/gradle-testing-base-7.3.3.jar)
/tmp/gradle_48.txt-WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
/tmp/gradle_48.txt-WARNING: System::setSecurityManager will be removed in a future release
/tmp/gradle_48.txt-WARNING: A terminally deprecated method in java.lang.System has been called
/tmp/gradle_48.txt-WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.3.3-all/4295vidhdd9hd3gbjyw1xqxpo/gradle-7.3.3/lib/plugins/gradle-testing-base-7.3.3.jar)
/tmp/gradle_48.txt-WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
/tmp/gradle_48.txt-WARNING: System::setSecurityManager will be removed in a future release
/tmp/gradle_48.txt-WARNING: A terminally deprecated method in java.lang.System has been called
/tmp/gradle_48.txt-WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.3.3-all/4295vidhdd9hd3gbjyw1xqxpo/gradle-7.3.3/lib/plugins/gradle-testing-base-7.3.3.jar)
/tmp/gradle_48.txt-WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
--
/tmp/gradle_6138.txt:REPRODUCE WITH: ./gradlew ':modules:transport-netty4:internalClusterTest' --tests "org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT.testLoggingHandler" -Dtests.seed=93CECD392A0BE99F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=bg -Dtests.timezone=Indian/Christmas -Druntime.java=17
/tmp/gradle_6138.txt-
/tmp/gradle_6138.txt-org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT > testLoggingHandler FAILED
/tmp/gradle_6138.txt-    java.lang.AssertionError:
/tmp/gradle_6138.txt-    Expected: an empty collection
/tmp/gradle_6138.txt-         but: <[org.apache.logging.log4j.spi.AbstractLogger caught java.lang.StackOverflowError logging ReusableSimpleMessage: [id: 0x8c3b0786, L:/127.0.0.1:42217 - R:/127.0.0.1:32856] WRITE: 30365B
/tmp/gradle_6138.txt-             +-------------------------------------------------+
/tmp/gradle_6138.txt-             |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
/tmp/gradle_6138.txt-    +--------+-------------------------------------------------+----------------+
/tmp/gradle_6138.txt-    |00000000| 45 53 00 00 76 97 00 00 00 00 00 00 00 1b 01 08 |ES..v...........|
/tmp/gradle_6138.txt-    |00000010| 1e 84 e3 00 00 00 02 00 00 07 6e 6f 64 65 5f 74 |..........node_t|
/tmp/gradle_6138.txt-    |00000020| 33 16 36 4b 56 77 56 5f 73 30 54 35 71 6c 71 54 |3.6KVwV_s0T5qlqT|
/tmp/gradle_6138.txt-    |00000030| 5f 43 6b 6b 75 77 62 51 16 4a 33 5f 52 42 6d 64 |_CkkuwbQ.J3_RBmd|
/tmp/gradle_6138.txt-    |00000040| 56 51 50 36 4a 38 57 36 4d 51 6c 6e 48 30 67 09 |VQP6J8W6MQlnH0g.|
/tmp/gradle_6138.txt-    |00000050| 31 32 37 2e 30 2e 30 2e 31 09 31 32 37 2e 30 2e |127.0.0.1.127.0.|
/tmp/gradle_6138.txt-    |00000060| 30 2e 31 04 7f 00 00 01 09 31 32 37 2e 30 2e 30 |0.1......127.0.0|
/tmp/gradle_6138.txt-    |00000070| 2e 31 00 00 a4 e9 01 1f 73 68 61 72 64 5f 69 6e |.1......shard_in|
/tmp/gradle_6138.txt-    |00000080| 64 65 78 69 6e 67 5f 70 72 65 73 73 75 72 65 5f |dexing_pressure_|
/tmp/gradle_6138.txt-    |00000090| 65 6e 61 62 6c 65 64 04 74 72 75 65 03 04 64 61 |enabled.true..da|
/tmp/gradle_6138.txt-    |000000a0| 74 61 01 64 01 06 69 6e 67 65 73 74 01 69 00 15 |ta.d..ingest.i..|
/tmp/gradle_6138.txt-    |000000b0| 72 65 6d 6f 74 65 5f 63 6c 75 73 74 65 72 5f 63 |remote_cluster_c|

--
/tmp/gradle_6155.txt:REPRODUCE WITH: ./gradlew ':modules:transport-netty4:internalClusterTest' --tests "org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT.testLoggingHandler" -Dtests.seed=3678AF710F20B372 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=hu-HU -Dtests.timezone=Atlantic/Faroe -Druntime.java=17
/tmp/gradle_6155.txt-
/tmp/gradle_6155.txt-org.opensearch.transport.netty4.OpenSearchLoggingHandlerIT > testLoggingHandler FAILED
/tmp/gradle_6155.txt-    java.lang.AssertionError:
/tmp/gradle_6155.txt-    Expected: an empty collection
/tmp/gradle_6155.txt-         but: <[org.apache.logging.log4j.spi.AbstractLogger caught java.lang.StackOverflowError logging ReusableSimpleMessage: [id: 0x90876c25, L:/127.0.0.1:46005 - R:/127.0.0.1:36494] WRITE: 27734B
/tmp/gradle_6155.txt-             +-------------------------------------------------+
/tmp/gradle_6155.txt-             |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
/tmp/gradle_6155.txt-    +--------+-------------------------------------------------+----------------+
/tmp/gradle_6155.txt-    |00000000| 45 53 00 00 6c 50 00 00 00 00 00 00 00 1d 01 08 |ES..lP..........|
/tmp/gradle_6155.txt-    |00000010| 1e 84 e3 00 00 00 02 00 00 07 6e 6f 64 65 5f 74 |..........node_t|
/tmp/gradle_6155.txt-    |00000020| 34 16 58 61 4e 4b 49 76 61 67 53 4e 57 57 37 71 |4.XaNKIvagSNWW7q|
/tmp/gradle_6155.txt-    |00000030| 7a 4d 69 6e 44 47 50 77 16 50 4e 43 6f 7a 30 43 |zMinDGPw.PNCoz0C|
/tmp/gradle_6155.txt-    |00000040| 6d 53 44 61 37 67 6d 4e 48 57 4e 44 47 4c 67 09 |mSDa7gmNHWNDGLg.|
/tmp/gradle_6155.txt-    |00000050| 31 32 37 2e 30 2e 30 2e 31 09 31 32 37 2e 30 2e |127.0.0.1.127.0.|
/tmp/gradle_6155.txt-    |00000060| 30 2e 31 04 7f 00 00 01 09 31 32 37 2e 30 2e 30 |0.1......127.0.0|
/tmp/gradle_6155.txt-    |00000070| 2e 31 00 00 b3 b5 01 1f 73 68 61 72 64 5f 69 6e |.1......shard_in|
/tmp/gradle_6155.txt-    |00000080| 64 65 78 69 6e 67 5f 70 72 65 73 73 75 72 65 5f |dexing_pressure_|
/tmp/gradle_6155.txt-    |00000090| 65 6e 61 62 6c 65 64 04 74 72 75 65 03 04 64 61 |enabled.true..da|
/tmp/gradle_6155.txt-    |000000a0| 74 61 01 64 01 06 69 6e 67 65 73 74 01 69 00 15 |ta.d..ingest.i..|
/tmp/gradle_6155.txt-    |000000b0| 72 65 6d 6f 74 65 5f 63 6c 75 73 74 65 72 5f 63 |remote_cluster_c|

@dreamer-89
Copy link
Member Author

dreamer-89 commented Jan 10, 2022 •
edited
Loading

gradle_48.txt
gradle_6155.txt

@reta
Copy link
Collaborator

reta commented Jan 10, 2022

@dreamer-89 this one is tracked under #1767

@dreamer-89
Copy link
Member Author

dreamer-89 commented Jan 11, 2022 •
edited
Loading

Ran tests in repeat mode with 500 count and was able to get SUITE timeout in 2 cases. The test output depicts that test waits 20mins (indefinitely) while waiting for OpenSearchRejectedExecutionException exception on bulk request on coordinator node. Link

[2022-01-12T02:27:38,001][INFO ][o.o.i.ShardIndexingPressureSettingsIT] [testShardIndexingPressureEnforcedEnabledDisabledSetting] Utility Sending requests ==> 3348
[2022-01-12T02:47:04,154][INFO ][o.o.i.ShardIndexingPressureSettingsIT] [testShardIndexingPressureEnforcedEnabledDisabledSetting] [ShardIndexingPressureSettingsIT#testShardIndexingPressureEnforcedEnabledDisabledSetting]: cleaning up after test
[2022-01-12T02:47:04,155][WARN ][o.o.e.NodeEnvironment    ] [testShardIndexingPressureEnforcedEnabledDisabledSetting] lock assertion failed

Test thread state from jstack output also depicts that test thread is in WAITING state (indefinite wait) for bulk request output.

"TEST-ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting-seed#[333D4CC320F3CC3F]" ID=202 WAITING on org.opensearch.common.util.concurrent.BaseFuture$Sync@503b347a
	at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
	- waiting on org.opensearch.common.util.concurrent.BaseFuture$Sync@503b347a
	at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
	at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
	at app//org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:267)
	at app//org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:99)
	at app//org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:69)
	at app//org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:50)
	at app//org.opensearch.index.ShardIndexingPressureSettingsIT.lambda$testShardIndexingPressureEnforcedEnabledDisabledSetting$2(ShardIndexingPressureSettingsIT.java:419)
	at app//org.opensearch.index.ShardIndexingPressureSettingsIT$$Lambda$4112/0x0000000801547068.run(Unknown Source)
	at app//org.apache.lucene.util.LuceneTestCase._expectThrows(LuceneTestCase.java:2879)
	at app//org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2754)
	at app//org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2749)
	at app//org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting(ShardIndexingPressureSettingsIT.java:419)
	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base@17.0.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@17.0.1/java.lang.reflect.Method.invoke(Method.java:568)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at app//org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at app//org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at app//org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at app//org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base@17.0.1/java.lang.Thread.run(Thread.java:833)

Gradle output:
Uploading gradle_91110.txt…

@reta
Copy link
Collaborator

reta commented Jan 11, 2022

Yeah, ShardIndexingPressureSettingsIT is problematic, we could change the Future::get to use timeout instead but the test will fail anyway :(

@reta reta mentioned this issue Jan 13, 2022
Closed
@andrross andrross mentioned this issue Jan 14, 2022
Closed
37 tasks
@dreamer-89
Copy link
Member Author

dreamer-89 commented Jan 15, 2022 •
edited
Loading

Added logs to get more insights when the timeout occurs. It seems the test is timing out because bulk request is not caught (bug?) as sharding indexing pressure breached (last successful duration limit) here by ShardIndexingMemoryManager. This bypass results in thread waiting for CountDonwLatch.await() call and thus timeout.

expectThrows(OpenSearchRejectedExecutionException.class, () -> client(coordinatingOnlyNode).bulk(bulkRequest).actionGet());

Commit
gradle_jan14_190.txt

@dreamer-89
Copy link
Member Author

The code is expected to fail the below check but it doesn't because of below condition including nodeTotalBytes and nodeLimit values.

if (((double) nodeTotalBytes / nodeLimit) < this.nodeSoftLimit)

To get why this check is failing; added another log statement to find out why this condition is not failing and raising OpenSearchRejectedExecutionException exception.

@dreamer-89
Copy link
Member Author

As expected, nodeTotalBytes value is not evaluated correctly by the time third requested started. This could possibly happen due to asyn nature of requests resulting in out of order requests.

[2022-01-16T08:39:37,866][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [testShardIndexingPressureEnforcedEnabledDisabledSetting] nodeTotalBytes ==> 22616, nodeLimit ==> 85940
[2022-01-16T08:39:37,866][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [testShardIndexingPressureEnforcedEnabledDisabledSetting] Soft limit not breached
[2022-01-16T08:39:37,867][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [testShardIndexingPressureEnforcedEnabledDisabledSetting] nodeTotalBytes ==> 45232, nodeLimit ==> 85940
[2022-01-16T08:39:37,867][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [testShardIndexingPressureEnforcedEnabledDisabledSetting] Soft limit not breached
[2022-01-16T08:39:37,869][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [node_t2] nodeTotalBytes ==> 45232, nodeLimit ==> 85940
[2022-01-16T08:39:37,869][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [node_t2] nodeTotalBytes ==> 22616, nodeLimit ==> 85940
[2022-01-16T08:39:37,869][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [node_t2] Soft limit not breached
[2022-01-16T08:39:37,869][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [node_t2] Soft limit not breached
[2022-01-16T08:39:37,893][INFO ][o.o.i.ShardIndexingPressureSettingsIT] [testShardIndexingPressureEnforcedEnabledDisabledSetting] coordinatingShardTracker.getCoordinatingOperationTracker().getPerformanceTracker().getTotalOutstandingRequests() ==> 0
[2022-01-16T08:39:37,894][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [testShardIndexingPressureEnforcedEnabledDisabledSetting] nodeTotalBytes ==> 45232, nodeLimit ==> 85940
[2022-01-16T08:39:37,894][INFO ][o.o.i.ShardIndexingPressureMemoryManager] [testShardIndexingPressureEnforcedEnabledDisabledSetting] Soft limit not breached
[2022-01-16T08:49:00,103][INFO ][o.o.i.ShardIndexingPressureSettingsIT] [testShardIndexingPressureEnforcedEnabledDisabledSetting] [ShardIndexingPressureSettingsIT#testShardIndexingPressureEnforcedEnabledDisabledSetting]: cleaning up after test

gradle_jan15_154.txt

@dreamer-89
Copy link
Member Author

There are couple of ways to fix this:

  1. Increase existing thread sleep time (current is 25ms) to 1sec here.
  2. Wait for previous two requests to be processed i.e. total outstanding requests to be 2 (better as not relying on hard coded wait time).

@dreamer-89 dreamer-89 mentioned this issue Jan 17, 2022
Merged
5 tasks
@dreamer-89
Copy link
Member Author

The failures were not observed post adding wait statement for outstanding requests to complete.

@dreamer-89
Copy link
Member Author

dreamer-89 commented Jan 17, 2022 •
edited
Loading

Summary

testShardIndexingPressureEnforcedEnabledDisabledSetting excecises different shard indexing pressure parameters such as outstanding request count, last successful request elapsed time shard indexing etc. The test is written based on assumption that outstanding requests will be processed on the coordinator node, before the test assertion; which sometimes fails. A simple fix is to wait for processing of outstanding requests.

What test does

To exercise different shard indexing pressure parameters, the test performs bulk requests in succession on the coordinator node. To have requests outstanding, a mock CountDownLatch behaviour is added on primary node transport; and thus builds the outstanding request count.

Broad set of steps in the test

  1. Set max outstanding requests to 1 and successful request elapsed timeout to 10ms
  2. Send batch of bulk requests:
    • First bulk request
    • Second bulk request
    • Third request is expected to throw OpenSearchRejectedExecutionException exception
  3. Disable enforced shard indexing pressure setting
  4. Repeat step 2 but don't expect exception because enforce setting is disabled.
  5. Enable enforcd shard indexing pressure setting
  6. Repeast step 2 but expect exception to be thrown on third request as setting enabled in step 5 above.

Debug

The issue was debugged with the help of log statements. Each time, logs were added one layer deep to get understanding why test is failing. The test were run multiple times in batch of 100/500 in order to repro the test timeout issue.

for num in {0..100}; do ./gradlew internalClusterTest >> /tmp/gradle_$num.txt 2>&1; cp server/build/reports/tests/internalClusterTest/classes/org.opensearch.index.ShardIndexingPressureSettingsIT.html /tmp/output_$num;   sleep 10; done;

Thanks @andrross for help throughout the task and @getsaurabh02 for providing shard indexing insights.

Learnings

  • Use of screen tool to conveniently run a command over nohup
  • Java thread dump analysis and tools to obtain thread state dumps on a running process and use of jps and jstack
  • Shard Indexing pressure settings innor working

Commit

Commit fixes the issue.
Will wait for review and couple of more tests runs before closing this issue.

@dreamer-89 dreamer-89 mentioned this issue Jan 19, 2022
Merged
5 tasks
@tlfeng tlfeng mentioned this issue Jan 21, 2022
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dreamer-89 dreamer-89

Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@reta @kotwanikunal @anasalkouz @dreamer-89