Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events #7732

Merged
merged 8 commits into from
May 26, 2023

Conversation

dreamer-89
Copy link
Member

@dreamer-89 dreamer-89 commented May 24, 2023

Description

  1. Use replica shards's ReplicationCheckpoint for replication events. This is a bug because RC is sent to primary which uses it to compare replica shard specific attributes.
  2. Use codec defined in engine config instead of default codec. This will break for cases where primary shard is not on default Lucene codec (e.g. Lucene95, Lucene94 etc).
  3. Driven from review comment: Removed ReplicationCheckpoint param from SegmentReplicationTarget ctor and SegmentReplicationTargetService.startReplication method.

Related Issues

Resolves: #7733
Related: #3881

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…heckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
Copy link
Member Author

Gradle Check (Jenkins) Run Completed with:

#7679

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

@codecov
Copy link

codecov bot commented May 25, 2023

Codecov Report

Merging #7732 (7970501) into main (70b109d) will increase coverage by 0.01%.
The diff coverage is 37.03%.

@@             Coverage Diff              @@
##               main    #7732      +/-   ##
============================================
+ Coverage     70.63%   70.65%   +0.01%     
+ Complexity    56099    56078      -21     
============================================
  Files          4680     4680              
  Lines        266083   266080       -3     
  Branches      39074    39074              
============================================
+ Hits         187955   188005      +50     
+ Misses        62187    62111      -76     
- Partials      15941    15964      +23     
Impacted Files Coverage Δ
...s/replication/SegmentReplicationTargetService.java 60.73% <36.00%> (+7.72%) ⬆️
.../indices/replication/SegmentReplicationTarget.java 90.26% <50.00%> (ø)

... and 485 files with indirect coverage changes

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

@dreamer-89 dreamer-89 merged commit adf7e2c into opensearch-project:main May 26, 2023
@dreamer-89
Copy link
Member Author

dreamer-89 commented May 26, 2023

The codecov violation is reported on code which changed due to refactor rather actual functionality test. Added unit tests for ForceSegmentSync handler to increase the coverage.

@dreamer-89 dreamer-89 added the backport 2.x Backport to 2.x branch label May 26, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request May 26, 2023
…heckpoint for replication events (#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
(cherry picked from commit adf7e2c)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
dreamer-89 pushed a commit that referenced this pull request May 26, 2023
…heckpoint for replication events (#7732) (#7769)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events



* Spotless fix



* Fix failing unit test



* Address review comments



* Address review comments rename



* Address review comments



* Add unit test for ForceSegmentSync transport handler



* Add unit test for ForceSegmentSync transport failure



---------


(cherry picked from commit adf7e2c)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.7 2.7
# Navigate to the new working tree
pushd ../.worktrees/backport-2.7
# Create a new branch
git switch --create backport/backport-7732-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 adf7e2cdfbc13013448bc07ddac0fe09c07fd7e5
# Push it to GitHub
git push --set-upstream origin backport/backport-7732-to-2.7
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport/backport-7732-to-2.7.

dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request May 26, 2023
…heckpoint for replication events (opensearch-project#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
suranjay pushed a commit to suranjay/OpenSearch that referenced this pull request May 29, 2023
…heckpoint for replication events (opensearch-project#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit that referenced this pull request May 30, 2023
…heckpoint for replication events (#7789)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events (#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix after merge conflict resolution

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
stephen-crawford pushed a commit to stephen-crawford/OpenSearch that referenced this pull request May 31, 2023
…heckpoint for replication events (opensearch-project#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
austintlee pushed a commit to austintlee/OpenSearch that referenced this pull request Jun 2, 2023
…heckpoint for replication events (opensearch-project#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
dreamer-89 added a commit to dreamer-89/OpenSearch that referenced this pull request Jun 2, 2023
…icationCheckpoint for replication events (opensearch-project#7732)"

This reverts commit adf7e2c.
gaiksaya pushed a commit to gaiksaya/OpenSearch that referenced this pull request Jun 26, 2023
…heckpoint for replication events (opensearch-project#7732) (opensearch-project#7769)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events



* Spotless fix



* Fix failing unit test



* Address review comments



* Address review comments rename



* Address review comments



* Add unit test for ForceSegmentSync transport handler



* Add unit test for ForceSegmentSync transport failure



---------


(cherry picked from commit adf7e2c)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…heckpoint for replication events (opensearch-project#7732)

* [Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Spotless fix

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing unit test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments rename

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport handler

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add unit test for ForceSegmentSync transport failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] [Segment Replication] Incorrect use of default codec and replication checkpoint
3 participants