Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] org.opensearch.discovery.SnapshotDisruptionIT.testDisruptionAfterShardFinalization intermittent failure #2062

Closed
dblock opened this issue Feb 7, 2022 · 1 comment
Closed

[BUG] org.opensearch.discovery.SnapshotDisruptionIT.testDisruptionAfterShardFinalization intermittent failure #2062

dblock opened this issue Feb 7, 2022 · 1 comment
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run

Comments

@dblock
Copy link
Member

dblock commented Feb 7, 2022 

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.discovery.SnapshotDisruptionIT.testDisruptionAfterShardFinalization" -Dtests.seed=10DB31204AEF8BC5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-Hant-HK -Dtests.timezone=America/Port-au-Prince -Druntime.java=17

org.opensearch.discovery.SnapshotDisruptionIT > testDisruptionAfterShardFinalization FAILED
    java.lang.AssertionError: Timeout waiting for node [node_t0] to be blocked
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.waitForBlock(AbstractSnapshotIntegTestCase.java:224)
        at org.opensearch.discovery.SnapshotDisruptionIT.testDisruptionAfterShardFinalization(SnapshotDisruptionIT.java:221)

    java.lang.IllegalStateException: Cannot cleanup [test-repo] - a snapshot is currently running in [SnapshotsInProgress[snapshot-2]]
        at org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$2.execute(TransportCleanupRepositoryAction.java:250)
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:63)
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:803)
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:378)
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:249)
        at org.opensearch.cluster.service.MasterService.access$000(MasterService.java:86)
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:173)
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:175)
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:213)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:275)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:238)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

    java.lang.AssertionError: Found leaked ending snapshots [test-repo:snapshot-2/6MQ9blNsRJ6qXqg2gze9ig] on [{node_t0}{DwXo2jCuQteamUgfRC0nDQ}{rOEB9z9NT1ONyVYUrf6ukg}{127.0.0.1}{127.0.0.1:36207}{m}{shard_indexing_pressure_enabled=true}]
        at org.opensearch.snapshots.SnapshotsService.assertAllListenersResolved(SnapshotsService.java:3196)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.lambda$verifyNoLeakedListeners$0(AbstractSnapshotIntegTestCase.java:140)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1048)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.verifyNoLeakedListeners(AbstractSnapshotIntegTestCase.java:138)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:833)

    SnapshotInProgressException[Cannot delete indices that are being snapshotted: [[test/oBsssOkRQZ69Q9rkpnR4Iw]]. Try again after snapshot finishes or cancel the currently running snapshot.]
        at org.opensearch.cluster.metadata.MetadataDeleteIndexService.deleteIndices(MetadataDeleteIndexService.java:136)
        at org.opensearch.cluster.metadata.MetadataDeleteIndexService$1.execute(MetadataDeleteIndexService.java:101)
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:63)
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:803)
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:378)
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:249)
        at org.opensearch.cluster.service.MasterService.access$000(MasterService.java:86)
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:173)
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:175)
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:213)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:275)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:238)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:833)

#1268 (comment)
@dblock dblock added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Feb 7, 2022
@dblock dblock mentioned this issue Feb 7, 2022
Merged
5 tasks
@Poojita-Raj Poojita-Raj mentioned this issue Nov 15, 2022
Closed
37 tasks
@dreamer-89
Copy link
Member

Closing this ticket as not failure instance reported in last 2 months. Used script to determine failure.
#1715 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dblock @anasalkouz @dreamer-89