[Segment Replication] Fix flaky testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion test

[dreamer-89]

Signed-off-by: Suraj Singh surajrider@gmail.com

Description

Fixes the flaky testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion. It does this by

• Replacing BackgroundIndexer with simple for loop as test was performing indexing operations twice which has chance of doc ids conflict
• Wait for replica shard to catch up with primary before doc count assertion in the end
• Assert on green state post snapshot restore

Test passes 500/500 locally

Issues Resolved

#5669

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10973/
• CommitID: fef30f3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationStatsIT.testSegmentReplicationStatsResponse

• URL: https://build.ci.opensearch.org/job/gradle-check/10975/
• CommitID: 741bbd3
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov-commenter]

Codecov Report

Merging #6280 (741bbd3) into main (1f4cdd2) will increase coverage by 0.08%.
The diff coverage is 39.10%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6280      +/-   ##
============================================
+ Coverage     70.72%   70.80%   +0.08%     
- Complexity    58791    58895     +104     
============================================
  Files          4782     4789       +7     
  Lines        281484   281788     +304     
  Branches      40642    40669      +27     
============================================
+ Hits         199072   199529     +457     
+ Misses        65984    65879     -105     
+ Partials      16428    16380      -48     

Impacted Files	Coverage Δ
...es/replication/SegmentReplicationStatsRequest.java	0.00% <0.00%> (ø)
...ication/SegmentReplicationStatsRequestBuilder.java	0.00% <0.00%> (ø)
.../org/opensearch/client/support/AbstractClient.java	32.00% <0.00%> (-0.31%)	⬇️
...java/org/opensearch/index/shard/StoreRecovery.java	65.74% <0.00%> (ø)
.../indices/replication/SegmentReplicationSource.java	100.00% <ø> (ø)
...cation/TransportSegmentReplicationStatsAction.java	11.11% <11.11%> (ø)
...s/replication/SegmentReplicationStatsResponse.java	13.33% <13.33%> (ø)
...h/indices/replication/SegmentReplicationState.java	46.96% <24.65%> (-31.98%)	⬇️
...s/replication/SegmentReplicationTargetService.java	50.00% <41.66%> (-0.72%)	⬇️
...ex/translog/transfer/TranslogTransferMetadata.java	91.83% <50.00%> (+2.74%)	⬆️
... and 510 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6280-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bcdf0c91df4ef748d07112c14be07b37bdf25792
# Push it to GitHub
git push --set-upstream origin backport/backport-6280-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6280-to-2.x.