DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock is flaky

[andrross]

See https://build.ci.opensearch.org/job/gradle-check/9659/

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock" -Dtests.seed=47FA74E827DEE212 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=th-TH -Dtests.timezone=Africa/Djibouti -Druntime.java=19

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testIndexCreateBlockWithAReadOnlyBlock FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([47FA74E827DEE212:57BCB87090067887]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$testIndexCreateBlockWithAReadOnlyBlock$11(DiskThresholdDeciderIT.java:297)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockWithAReadOnlyBlock(DiskThresholdDeciderIT.java:295)

This was introduced in #5852. Maybe this is what #5952 is trying to fix? @RS146BIJAY

[RS146BIJAY]

Yes. It seems like the test cases are flaky even now. I am trying to fix those flaky test cases as a part of #5952 .

[sachinpkale]

Build failure due to these tests - https://build.ci.opensearch.org/job/gradle-check/9749/

[andrross]

Seems like this isn't completely fixed: https://build.ci.opensearch.org/job/gradle-check/10405/

/cc @RS146BIJAY

[RS146BIJAY]

Unable to reproduce this failure on local. Validating what else may be causing race condition scenario which is causing test cases to fail.

[RS146BIJAY]

Seems race condition is happening between data populated on the cluster and the actual free space reflected by the file system. Removing the indexing call and explicitly setting free and total space as done inside MockDiskUsagesIT seems to fix this race condition.

[andrross]

I believe this was fixed by #6277. Will reopen if it reoccurs.