[BUG] Gradle check is unreliable

[peternied]

Describe the bug

A pull request #12382 was created on the evening of 2/19, with no code changes, it only modified a markdown file. The gradle check failed 6 times in a row. Each attempt took an hour or more, preventing this pull request from merging for a full day after it was approved by multiple maintainers.

Related component

Build

To Reproduce

1. Create a new PR
2. Monitor how often gradle check fails.

Expected behavior

Gradle check should not fail due to changes outside the scope of the pull request.

Additional Details

Flaky test round up

Batch 1

• [BUG] org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction #12197

Batch 2

• [BUG] org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock flaky #10006
• [BUG] org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing flaky failure #9191

Batch 3

• [BUG] org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing flaky failure #9191
• [BUG] the org.opensearch.snapshots.DeleteSnapshotIT tests are flaky #9208

Batch 4

• [BUG] IndexActionIT tests are flaky Shard [test][0] is still locked after 5 sec waiting #12408

Batch 5

• [BUG] org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction #12197
• [BUG] org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing flaky failure #9191
• [BUG] QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown is flaky #11979
• [BUG] the org.opensearch.snapshots.DeleteSnapshotIT tests are flaky #9208

Batch 6

• [BUG] org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing flaky failure #9191
• [BUG] SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled is flaky #12409

Batch 7

• [BUG] org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing flaky failure #9191

[reta]

@peternied the flaky tests impact every single build, w/o code, I am not sure we need yet another issue on that, please share your thoughts #11217, thank you.

[peternied]

For many Amazon developers there was an initiative to reduce the number of flaky test starting in January until this release. I'm not sure where that effort landed, but looking at this issue gives me pause. I filed this issue because it accounts for the actual toll on a contribution rather than an abstract scenario.

[peternied]

[Triage - attendees 1 2 3 4 5]
We are accepting this issue as a focal point for discussion.

Related:

• [Meta] gradle check improvements #4053