[CI] flaky test failure - o.o.indices.stats.IndexStatsIT.testFilterCacheStats

[tlfeng]

Caught from PR #2424 (comment):

❌ Gradle Check failure 90f143a
Log 3504

Reports 3504

In log 3504:

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.stats.IndexStatsIT.testFilterCacheStats" -Dtests.seed=D48688166EC0CB86 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr -Dtests.timezone=GMT -Druntime.java=17

org.opensearch.indices.stats.IndexStatsIT > testFilterCacheStats FAILED
    java.lang.AssertionError: expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([D48688166EC0CB86:B1B97553DEC8CCFF]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.indices.stats.IndexStatsIT.testFilterCacheStats(IndexStatsIT.java:1225)

[andrross]

Another failure: https://build.ci.opensearch.org/job/gradle-check/6332/

[dblock]

#3633

[andrross]

#5365

[Rishikesh1159]

#5332

[mch2]

I've gotten this to reproduce locally. Its failing on this line that asserts there is nothing in the query cache. At this point in the test no searches have been initiated from the test but there is occasionally an entry in the cache.

Up until that point in the test the only reqs made are to index a random set of docs. Looks like a query is being made somewhere from inside engine on the index path that reuses the same querycache, I think that is here where only if assertions are enabled we check if the doc is not actually in the index from indexIntoLucene. I don't think this is related to randomization, but that the query is only occasionally cached or evicted before the test fetches stats.

I've nulled out / used DisabledQueryCache impl on this searcher and am no longer able to repro the failure.