[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch #4181

Rishikesh1159 · 2022-08-09T23:28:47Z

Description

This PR backports following 5 PR's from main branch:

[Segment Replication] Add components for segment replication to perform file copy. #3525.
Add replication orchestration for a single shard #3533.
Adding onNewCheckpoint to Start Replication on Replica Shard when Segment Replication is turned on #3540.
[Segment Replication] Fixing flaky test failure happening for testShardAlreadyReplicating() #3943
[Segment Replication] Fix possible flaky test for testBeforeIndexShardClosed_CancelsOngoingReplications() #3963

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…oject#3525. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

github-actions · 2022-08-09T23:41:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1639/
CommitID: d979fd2

…rdAlreadyReplicating() (opensearch-project#3943) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

…Replications() (opensearch-project#3963) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

…ard class. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

github-actions · 2022-08-10T05:19:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1647/
CommitID: 4b60a68

codecov-commenter · 2022-08-10T05:21:51Z

Codecov Report

Merging #4181 (7f6da3f) into 2.x (7062283) will increase coverage by 0.01%.
The diff coverage is 76.64%.

@@             Coverage Diff              @@
##                2.x    #4181      +/-   ##
============================================
+ Coverage     70.45%   70.46%   +0.01%     
- Complexity    56690    56769      +79     
============================================
  Files          4563     4569       +6     
  Lines        273024   273348     +324     
  Branches      40059    40083      +24     
============================================
+ Hits         192364   192628     +264     
- Misses        64521    64524       +3     
- Partials      16139    16196      +57

Impacted Files	Coverage Δ
.../main/java/org/opensearch/index/engine/Engine.java	`73.70% <ø> (+0.48%)`	⬆️
...va/org/opensearch/index/engine/InternalEngine.java	`74.17% <ø> (+0.88%)`	⬆️
...rg/opensearch/indices/recovery/RecoveryTarget.java	`70.80% <ø> (-0.03%)`	⬇️
...search/indices/recovery/RecoveryTargetHandler.java	`100.00% <ø> (ø)`
...rch/indices/recovery/RetryableTransportClient.java	`81.81% <ø> (ø)`
.../replication/checkpoint/ReplicationCheckpoint.java	`42.42% <0.00%> (+6.06%)`	⬆️
...ckpoint/SegmentReplicationCheckpointPublisher.java	`100.00% <ø> (ø)`
...replication/common/ReplicationFailedException.java	`12.50% <12.50%> (ø)`
...s/replication/SegmentReplicationSourceService.java	`56.81% <44.00%> (-24.27%)`	⬇️
...s/replication/SegmentReplicationTargetService.java	`71.92% <60.00%> (-6.12%)`	⬇️
... and 498 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Rishikesh1159 · 2022-08-10T05:33:43Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -2657,7 +2708,6 @@ public long getLocalCheckpoint() {
     * Also see {@link #getLocalCheckpoint()}.
     */
    public long getProcessedLocalCheckpoint() {
-        assert indexSettings.isSegRepEnabled();


This assert has been removed because it is failing many unit tests in SegmentReplicationTargetServiceTests.java and SegmentReplicationIndexShardTests.java. The unit tests in these classes don't use index setting as SEGMENT, so when an assert is done on if segRepEnabled() all of the shards fails the assert and don't perform necessary unit tests.

This assert has been added to 2.x branch in this backport PR. As the original PR had method getProcessedLocalCheckpoint() in Engine.java as abstract which is a breaking. To avoid breaking breaking changes we made few changes to original PR while backporting. And this assert is part of those changes. It is not necessary to have this assert as it will any how be removed in 3.0 opensearch

I added the SEGMENT index setting in test classes. Can we backport #4163 if it helps ?

I think it might help actually, but let me check by backporting your PR locally and add back assert statement to see if all of the unit tests pass

I tried adding back assert and use SEGMENT replication as index setting in unit tests. But got this error:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17 2> java.lang.AssertionError at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0) at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711) at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

so there are multiple places we have to makes changes to make this work and I don't think this PR is right place to do all changes. We can do them while backporting other PR's. So for now I am removing assert statement

…ation in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

github-actions · 2022-08-10T17:39:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1663/
CommitID: 8c5753b

Rishikesh1159 · 2022-08-10T17:49:36Z

Flaky test:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0)
        at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711)
        at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

Actually this is not flaky test. this happened due to changes in recent commit. Will fix this in next commit

dreamer-89 · 2022-08-10T17:51:20Z

Flaky test:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0)
        at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711)
        at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

Another appearance in #4041. Created #4184 to track it as it is failing more oftenly on CI.

github-actions · 2022-08-10T18:10:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/1664/
CommitID: 8c5753b

Rishikesh1159 · 2022-08-10T18:25:48Z

Flaky test:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.OngoingSegmentReplicationsTests.testMultipleReplicasUseSameCheckpoint" -Dtests.seed=A0971E691B2879E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=Asia/Kolkata -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([A0971E691B2879E0:A4046FDAE5B7632]:0)
        at org.opensearch.index.shard.IndexShard.getProcessedLocalCheckpoint(IndexShard.java:2711)
        at org.opensearch.indices.replication.OngoingSegmentReplicationsTests.setUp(OngoingSegmentReplicationsTests.java:73)

Another appearance in #4041. Created #4184 to track it as it is failing more oftenly on CI.

Sorry @dreamer-89 , I thought this is a flaky test but not. This happened because of adding "assert statement" back that we discussed in above comments. I will remove the assert to fix this

…t replication in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests." Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> This reverts commit 8c5753b.

github-actions · 2022-08-10T19:37:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/1667/
CommitID: 7f6da3f

mch2 and others added 4 commits August 9, 2022 22:51

Resolving import conflict in Node.java and mergining PR opensearch-pr…

48e8ade

…oject#3525. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Resolving conflicts and merging PR opensearch-project#3533.

6cffde4

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Resolving conflicts and Merging PR opensearch-project#3540.

9b83b52

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Applying spotlesscheck and fixing wildcard imports.

d979fd2

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Rishikesh1159 requested review from a team and reta as code owners August 9, 2022 23:28

Rishikesh1159 and others added 3 commits August 10, 2022 01:26

[Segment Replication] Fixing flaky test failure happening for testSha…

8ea270f

…rdAlreadyReplicating() (opensearch-project#3943) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Fix possible flaky test for testBeforeIndexShardClosed_CancelsOngoing…

24c2e0b

…Replications() (opensearch-project#3963) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Removing assert segrep() in getProcessedLocalCheckpoint() of Index.sh…

4b60a68

…ard class. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Rishikesh1159 changed the title ~~[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 From main branch~~ [Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch Aug 10, 2022

Rishikesh1159 commented Aug 10, 2022

View reviewed changes

Adding back assert statement and make index setting to segment replic…

8c5753b

…ation in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

dreamer-89 mentioned this pull request Aug 10, 2022

[BUG] [Segment Replication] Flaky test SegmentReplicationTargetServiceTests.testReplicationOnDone #4184

Closed

Revert "Adding back assert statement and make index setting to segmen…

7f6da3f

…t replication in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests." Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> This reverts commit 8c5753b.

tlfeng approved these changes Aug 11, 2022

View reviewed changes

tlfeng added backport PRs or issues specific to backporting features or enhancments v2.3.0 'Issues and PRs related to version v2.3.0' feature New feature or request distributed framework labels Aug 11, 2022

Rishikesh1159 merged commit 97113b9 into opensearch-project:2.x Aug 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch #4181

[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch #4181

Rishikesh1159 commented Aug 9, 2022 •

edited

Loading

github-actions bot commented Aug 9, 2022

github-actions bot commented Aug 10, 2022

codecov-commenter commented Aug 10, 2022 •

edited

Loading

Rishikesh1159 Aug 10, 2022 •

edited

Loading

dreamer-89 Aug 10, 2022

Rishikesh1159 Aug 10, 2022 •

edited

Loading

Rishikesh1159 Aug 10, 2022

github-actions bot commented Aug 10, 2022

Rishikesh1159 commented Aug 10, 2022 •

edited

Loading

dreamer-89 commented Aug 10, 2022 •

edited

Loading

github-actions bot commented Aug 10, 2022

Rishikesh1159 commented Aug 10, 2022 •

edited

Loading

github-actions bot commented Aug 10, 2022

[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch #4181

[Backport 2.x] [Segment Replication] Backport PR's : #3525 #3533 #3540 #3943 #3963 From main branch #4181

Conversation

Rishikesh1159 commented Aug 9, 2022 • edited Loading

Description

github-actions bot commented Aug 9, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Aug 10, 2022 • edited Loading

Codecov Report

Rishikesh1159 Aug 10, 2022 • edited Loading

Choose a reason for hiding this comment

dreamer-89 Aug 10, 2022

Choose a reason for hiding this comment

Rishikesh1159 Aug 10, 2022 • edited Loading

Choose a reason for hiding this comment

Rishikesh1159 Aug 10, 2022

Choose a reason for hiding this comment

github-actions bot commented Aug 10, 2022

Gradle Check (Jenkins) Run Completed with:

Rishikesh1159 commented Aug 10, 2022 • edited Loading

dreamer-89 commented Aug 10, 2022 • edited Loading

github-actions bot commented Aug 10, 2022

Gradle Check (Jenkins) Run Completed with:

Rishikesh1159 commented Aug 10, 2022 • edited Loading

github-actions bot commented Aug 10, 2022

Gradle Check (Jenkins) Run Completed with:

Rishikesh1159 commented Aug 9, 2022 •

edited

Loading

codecov-commenter commented Aug 10, 2022 •

edited

Loading

Rishikesh1159 Aug 10, 2022 •

edited

Loading

Rishikesh1159 Aug 10, 2022 •

edited

Loading

Rishikesh1159 commented Aug 10, 2022 •

edited

Loading

dreamer-89 commented Aug 10, 2022 •

edited

Loading

Rishikesh1159 commented Aug 10, 2022 •

edited

Loading