[Remote Segment Store] Combine metadata and snapshot files

[sachinpkale]

Description

• Currently, we upload 2 files in addition to segment files per refresh (we upload these files even if there are no segments to upload - that will be fixed by [Remote Segments] Avoid segments metadata upload if no difference in local store and remote store #7443)
• These 2 files are metadata file and SegmentInfosSnapshot file. In order to avoid explosion of number of files in the remote store, we use the same name of metadata and snapshot file per commit.
• But the assumed invariant is: there is 1:1 mapping between metadata and SegmentInfosSnapshot file.
• This does not work if while reading we read metadata file and by the time we read snapshot file, remote store has new snapshot file with the same name.
• To avoid this race condition, this change combines these 2 files. metadata file will have one more field: segmentInfosBytes and in the restore flow, instance of SegmentInfos will be created by this field in the metadata.

Related Issues

• [Remote Segment Store] Combine metadata and SegmentInfosSnapshot files #7772

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16374/
• CommitID: b5eb9cc
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16376/
• CommitID: 44eb7d9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16384/
• CommitID: 4708e7a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/16408/
• CommitID: e8a9fb0
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7777 (71c24b4) into main (aa21585) will increase coverage by 0.06%.
The diff coverage is 96.92%.

@@             Coverage Diff              @@
##               main    #7777      +/-   ##
============================================
+ Coverage     70.78%   70.84%   +0.06%     
- Complexity    56251    56275      +24     
============================================
  Files          4689     4689              
  Lines        266304   266324      +20     
  Branches      39086    39087       +1     
============================================
+ Hits         188505   188682     +177     
+ Misses        61847    61664     -183     
- Partials      15952    15978      +26     

Impacted Files	Coverage Δ
...in/java/org/opensearch/index/shard/IndexShard.java	70.51% <86.66%> (+1.08%)	⬆️
...search/index/shard/RemoteStoreRefreshListener.java	85.05% <100.00%> (+0.10%)	⬆️
...earch/index/store/RemoteSegmentStoreDirectory.java	93.22% <100.00%> (+0.41%)	⬆️
...x/store/remote/metadata/RemoteSegmentMetadata.java	100.00% <100.00%> (ø)
.../remote/metadata/RemoteSegmentMetadataHandler.java	100.00% <100.00%> (ø)

... and 434 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16411/
• CommitID: 76dbbb7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16888/
• CommitID: a8de20f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/16889/
• CommitID: 81f60d5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/16895/
• CommitID: 611938b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[sachinpkale]

Precommit is failing on windows. Re-triggering the build.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

• URL: https://build.ci.opensearch.org/job/gradle-check/16897/
• CommitID: 71c24b4
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7777-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8dd5461bff681238546f08f91b49d6d0755abcc7
# Push it to GitHub
git push --set-upstream origin backport/backport-7777-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7777-to-2.x.