[BUG] Flakey test failure in SortedSetDVStringFieldDataTests testSortMissingLast

[kartg]

Describe the bug

First observed in #4183 (comment), but I hit the same failure several times in #4222 and was able to reproduce it locally:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.fielddata.SortedSetDVStringFieldDataTests.testSortMissingLast" -Dtests.seed=BC7C1431A4F70C5D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ga-IE -Dtests.timezone=Europe/Belgrade -Druntime.java=17

org.opensearch.index.fielddata.SortedSetDVStringFieldDataTests > testSortMissingLast FAILED
    java.lang.AssertionError: expected:<1105> but was:<1023>

To Reproduce
See above

Expected behavior
Test consistently passes

Plugins
None

Screenshots
N/A

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context

[kartg]

@nknize are you poking at this?

[dreamer-89]

Multiple occurrences in recent PRs:
#4237
#4221

[nknize]

Yes. It's new as of the Lucene 9.4 snapshot PR. Only reproducible if you run the entire Test class, not the individual case.

[owaiskazi19]

#4066

[dbwiddis]

Some more investigation. The number of documents in the search impacts the failure.

The failures occurs in edge cases where higher indexed elements in the document list are null. In the repro case listed above, there are 1105 documents, but the last one is null, and any number of documents 1103 or below fails (both 1104 and 1105 succeed).

The difference seems to align with the number of nulls in indices > 1000.

[dbwiddis]

From Lucene's IndexSearcher javaDoc:

NOTE: The search(org.apache.lucene.search.Query, int) and searchAfter(org.apache.lucene.search.ScoreDoc, org.apache.lucene.search.Query, int) methods are configured to only count top hits accurately up to 1,000 and may return a lower bound of the hit count if the hit count is greater than or equal to 1,000. On queries that match lots of documents, counting the number of hits may take much longer than computing the top hits so this trade-off allows to get some minimal information about the hit count without slowing down search too much. The TopDocs.scoreDocs array is always accurate however.

This conflicts with the assertion here:

OpenSearch/server/src/test/java/org/opensearch/index/fielddata/AbstractStringFieldDataTestCase.java

Line 285 in 05a5819

assertEquals(numDocs, topDocs.totalHits.value);

The value of topDocs.totalHits.value is a lower bound if it is above 1000. Whether this is the case can be determined using the totalHits.relation enum.

This test needs to be adjusted to account for this possible inaccuracy.