[BUG] org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository is flaky

[reta]

Describe the bug

The test case org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository is flaky:

org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository

java.lang.AssertionError: max seq. no. [1] does not match [0]
	at __randomizedtesting.SeedInfo.seed([5705DA441D4A7808]:0)
	at org.opensearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:202)
	at org.opensearch.index.engine.ReadOnlyEngine.ensureMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:189)
	at org.opensearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:140)
	at org.opensearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:76)
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2474)
	at org.opensearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2416)
	at org.opensearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2385)
	at org.opensearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:660)
	at org.opensearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:121)
	at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344)
	at org.opensearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:119)
	at org.opensearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2759)
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Storage:Remote

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteRestoreSnapshotIT.testRestoreShallowSnapshotRepository" -Dtests.seed=5705DA441D4A7808

Expected behavior

The test must always pass

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
Standard

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/32711/testReport/junit/org.opensearch.remotestore/RemoteRestoreSnapshotIT/testRestoreShallowSnapshotRepository/

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@reta Thanks for creating this issue

[reta]

Closing in favour of #14324