Fix boundary condition in indexing pressure test

[andrross]

This updates the boundary condition in an assertion in two tests in ShardIndexingPressureConcurrentExecutionTests. I could reliably reproduce errors here by running:

./gradlew ':server:test' -Dtests.iters=10000 --tests "org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testReplicaThreadedUpdateToShardLimits"

On every error the value that failed was exactly 0.95 and failed the less than check. The change here is to accept 0.95, and also refactor the test to give a better error message on failure.

Issues Resolved

Closes #2241
Closes #4215

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[andrross]

@getsaurabh02 To be honest I don't fully understand this assertion. Can you verify it is safe/correct to change this assertion from "less than 0.95" to "less than or equal to 0.95"?

[getsaurabh02]

The additional memory allocation per shard is governed by its peak memory utilization threshold of the current allocation. This is configurable setting and defaults to 95%.

Once the current allocation is greater than this threshold, the new shard limits are calculated and allocated. Since this condition checks for current value to be greater than, it should be safe to assume that the current limit can reach until the 95% (0.95) utilization. Hence "less than or equal to 0.95" should be good.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6662/
• CommitID: 07507c8
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6667/
• CommitID: e61fbac
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[getsaurabh02]

The additional memory allocation per shard is governed by its peak memory utilization threshold of the current allocation. This is configurable setting and defaults to 95%.

Once the current allocation is greater than this threshold, the new shard limits are calculated and allocated. Since this condition checks for current value to be greater than, it should be safe to assume that the current limit can reach until the 95% (0.95) utilization. Hence "less than or equal to 0.95" should be good.

[andrross]

Ironically a different shard indexing pressure test failed:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites" -Dtests.seed=567CA44C57BDC19B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr-Latn-ME -Dtests.timezone=Pacific/Palau -Druntime.java=19

org.opensearch.index.ShardIndexingPressureIT > testShardIndexingPressureTrackingDuringBulkWrites FAILED
    java.lang.AssertionError: 
    Expected: <22616L>
         but: was <22954L>
        at __randomizedtesting.SeedInfo.seed([567CA44C57BDC19B:DC58B8B44B77E953]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites(ShardIndexingPressureIT.java:209)

#5176

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/6688/
• CommitID: e61fbac
  Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6694/
• CommitID: 9f6d53a

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/6695/
• CommitID: 179c1e7