[Segment Replication] Checkpoint Replay on Replica Shard

[Rishikesh1159]

Description

This PR will allow us to store latest received checkpoint for replica shard and replay the latest checkpoint after previous checkpoint replication process is done.

This is a part of the process of merging our feature branch - feature/segment-replication - back into main by re-PRing our changes from the feature branch.
The breakdown of the plan to merge to main is detailed here: #2355
For added context on segment replication - here's the design proposal #2229

Issues Resolved

#3558

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-ci-bot]

❌   Gradle Check failure fcdfbd1
Log 6240

Reports 6240

[Rishikesh1159]

start gradle check

[opensearch-ci-bot]

✅   Gradle Check success fcdfbd1
Log 6243

Reports 6243

[kartg]

I'm a little concerned that this recursive call could lead to a long-running thread and/or function stack overflow. As a follow-up to this PR, we should look into wrapping checkpoint processing in a Runnable so we can execute it async from the new-checkpoint notification.

cc @mch2

[mch2]

any reason not to do this now? @Rishikesh1159 @kartg ? We had a similar implementation in the feature branch - https://github.com/opensearch-project/OpenSearch/blob/feature/segment-replication/server/src/main/java/org/opensearch/index/shard/IndexShard.java#L3683

[kartg]

Ah, i didn't realize the POC had the threading in place. Yes, if we have prior art, we can pull it into this PR

[Bukhtawar]

QQ: Do newer checkpoints supersede older checkpoints? What happens when the replication is slow is backlogged with too many checkpoints to process.

[Rishikesh1159]

Sure, I wrapped checkpoint processing in a Runnable in new commit

[Rishikesh1159]

@Bukhtawar we only store one latest received checkpoint and replace it whenever there is a new checkpoint. We don't store all received checkpoints to replica shard. So, even when replication is slow, we won't be backlogged with too many checkpoints, because we process only one checkpoint (which is latest received checkpoint)

[Bukhtawar]

Thanks and I am assuming replication usually happens on the generic threadpool to start with?

[Bukhtawar]

@Rishikesh1159, @mch2 I would recommend doing explicit checks rather than simply forking to guarantee we don't end up using the same thread

void execute(Runnable runnable, Thread originalThread) {
            if (originalThread == Thread.currentThread()) {
                fork(runnable);
            } else {
                runnable.run();
            }
        } 

[kartg]

Self-resolved one of my earlier comments, and the others are nitpicks.

[Bukhtawar]

Thanks @Rishikesh1159 left a comment on explicit checks on thread before we fork

[dreamer-89]

@Rishikesh1159 : Will be good to have test cases for failure scenario, multiple checkpoints and IT exercising this code.

[Rishikesh1159]

We have already covered failure scenario in previous PR.For multiple checkpoints we are already passing 2 checkpoints in many of our tests, so not sure if there is need to multiple checkpoints test case, But if needed we can do that also. And for IT for this, yes it is good to have an IT for this code, but I think we can make a different PR for that

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/818/
• CommitID: e0ec134

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/820/
• CommitID: 57afbd6

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/821/
• CommitID: 4322d35

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ABORTED ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/824/
• CommitID: 40c454c

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ABORTED ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/826/
• CommitID: 40c454c

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ABORTED ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/831/
• CommitID: 40c454c

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ABORTED ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/849/
• CommitID: 40c454c

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/869/
• CommitID: 40c454c