Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10725

mch2 · 2023-10-19T00:15:31Z

Description

This change makes Make RemoteStoreReplicationSource#getSegmentFiles asynchronous.
It also ensures that SegmentReplicationTargets are properly closed during cancellation with remote store enabled if the cancel is issued during a file download.

Related Issues

Resolves #8089
Resolves #10619

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-10-19T00:46:59Z

Compatibility status:

Checks if related components are compatible with change cd06a0d

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git]

github-actions · 2023-10-19T18:09:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28470/
CommitID: f5519ec
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-19T19:03:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28476/
CommitID: f5519ec
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Also make the remote store download process cancellable in case the replication event is canceled. Signed-off-by: Andrew Ross <andrross@amazon.com>

… metadata and segment fetch steps. Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2023-10-19T20:33:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28486/
CommitID: cd06a0d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

mch2 · 2023-10-19T20:42:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/28486/

CommitID: cd06a0d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

#6090

github-actions · 2023-10-19T21:25:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28498/
CommitID: cd06a0d

codecov · 2023-10-19T21:26:32Z

Codecov Report

Merging #10725 (cd06a0d) into main (69f6f4e) will increase coverage by 0.04%.
Report is 2 commits behind head on main.
The diff coverage is 77.19%.

@@             Coverage Diff              @@
##               main   #10725      +/-   ##
============================================
+ Coverage     71.11%   71.15%   +0.04%     
- Complexity    58530    58535       +5     
============================================
  Files          4854     4854              
  Lines        276049   276066      +17     
  Branches      40168    40170       +2     
============================================
+ Hits         196311   196448     +137     
+ Misses        63385    63214     -171     
- Partials      16353    16404      +51

Files	Coverage Δ
.../indices/replication/common/ReplicationTarget.java	`80.00% <100.00%> (+1.05%)`	⬆️
...ain/java/org/opensearch/search/DocValueFormat.java	`59.17% <0.00%> (-5.33%)`	⬇️
...nsearch/index/store/RemoteStoreFileDownloader.java	`92.15% <84.00%> (-0.87%)`	⬇️
...ices/replication/RemoteStoreReplicationSource.java	`85.24% <75.00%> (-5.83%)`	⬇️

... and 437 files with indirect coverage changes

andrross

This looks good to me! But I authored one of the commits in this PR so I'd love to have approval from another maintainer.

server/src/main/java/org/opensearch/indices/replication/common/ReplicationTarget.java

* Make RemoteStoreReplicationSource#getSegmentFiles asynchronous Also make the remote store download process cancellable in case the replication event is canceled. Signed-off-by: Andrew Ross <andrross@amazon.com> * Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps. Signed-off-by: Marc Handalian <handalm@amazon.com> * Wrap metadata fetch in cancellableThreads.executeIO Signed-off-by: Marc Handalian <handalm@amazon.com> * self review Signed-off-by: Marc Handalian <handalm@amazon.com> * spotless Signed-off-by: Marc Handalian <handalm@amazon.com> * Add missing node settings when bootstrapping nodes in tests. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit 41a12e2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… (#10771) * Make RemoteStoreReplicationSource#getSegmentFiles asynchronous Also make the remote store download process cancellable in case the replication event is canceled. * Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps. * Wrap metadata fetch in cancellableThreads.executeIO * self review * spotless * Add missing node settings when bootstrapping nodes in tests. --------- (cherry picked from commit 41a12e2) Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…earch-project#10725) * Make RemoteStoreReplicationSource#getSegmentFiles asynchronous Also make the remote store download process cancellable in case the replication event is canceled. Signed-off-by: Andrew Ross <andrross@amazon.com> * Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps. Signed-off-by: Marc Handalian <handalm@amazon.com> * Wrap metadata fetch in cancellableThreads.executeIO Signed-off-by: Marc Handalian <handalm@amazon.com> * self review Signed-off-by: Marc Handalian <handalm@amazon.com> * spotless Signed-off-by: Marc Handalian <handalm@amazon.com> * Add missing node settings when bootstrapping nodes in tests. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…earch-project#10725) * Make RemoteStoreReplicationSource#getSegmentFiles asynchronous Also make the remote store download process cancellable in case the replication event is canceled. Signed-off-by: Andrew Ross <andrross@amazon.com> * Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps. Signed-off-by: Marc Handalian <handalm@amazon.com> * Wrap metadata fetch in cancellableThreads.executeIO Signed-off-by: Marc Handalian <handalm@amazon.com> * self review Signed-off-by: Marc Handalian <handalm@amazon.com> * spotless Signed-off-by: Marc Handalian <handalm@amazon.com> * Add missing node settings when bootstrapping nodes in tests. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

This comment was marked as outdated.

Sign in to view

github-actions bot added distributed framework enhancement Enhancement or improvement to existing feature or request Indexing:Replication Issues and PRs related to core replication framework eg segrep labels Oct 19, 2023

mch2 added the skip-changelog label Oct 19, 2023

mch2 marked this pull request as ready for review October 19, 2023 04:59

mch2 requested a review from VachaShah as a code owner October 19, 2023 04:59

This comment was marked as outdated.

Sign in to view

mch2 force-pushed the parallel-download-async-v2 branch from 5f4e916 to 02172cc Compare October 19, 2023 16:11

This comment was marked as outdated.

Sign in to view

andrross and others added 6 commits October 19, 2023 12:47

Make RemoteStoreReplicationSource#getSegmentFiles asynchronous

b050e2e

Also make the remote store download process cancellable in case the replication event is canceled. Signed-off-by: Andrew Ross <andrross@amazon.com>

Add ITs ensuring segRep targets are cleaned up on cancellation during…

74d1548

… metadata and segment fetch steps. Signed-off-by: Marc Handalian <handalm@amazon.com>

Wrap metadata fetch in cancellableThreads.executeIO

1be22a3

Signed-off-by: Marc Handalian <handalm@amazon.com>

self review

3ac491f

Signed-off-by: Marc Handalian <handalm@amazon.com>

spotless

6416923

Signed-off-by: Marc Handalian <handalm@amazon.com>

Add missing node settings when bootstrapping nodes in tests.

cd06a0d

Signed-off-by: Marc Handalian <handalm@amazon.com>

mch2 force-pushed the parallel-download-async-v2 branch from f5519ec to cd06a0d Compare October 19, 2023 19:47

andrross reviewed Oct 19, 2023

View reviewed changes

github-actions bot added the bug Something isn't working label Oct 19, 2023

kotwanikunal approved these changes Oct 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/indices/replication/common/ReplicationTarget.java Show resolved Hide resolved

mch2 merged commit 41a12e2 into opensearch-project:main Oct 20, 2023
24 checks passed

mch2 added the backport 2.x Backport to 2.x branch label Oct 20, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 20, 2023

[Backport 2.x] Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10771

Merged

mch2 mentioned this pull request Nov 14, 2023

[Segment Replication] Add cancellation support in RemoteStoreReplicationSource #9234

Closed

6 tasks

This was referenced Jan 5, 2024

[BUG] RemoteIndexShardTests.testSegRepSucceedsOnPreviousCopiedFiles is flaky - suite timeout was reached #10885

Closed

Use SegmentReplicationTarget test to validate non-active on-disk files are reused for replication #11786

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10725

Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10725

mch2 commented Oct 19, 2023 •

edited

Loading

This comment was marked as outdated.

github-actions bot commented Oct 19, 2023 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

mch2 commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

codecov bot commented Oct 19, 2023

andrross left a comment

Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10725

Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10725

Conversation

mch2 commented Oct 19, 2023 • edited Loading

Description

Related Issues

Check List

This comment was marked as outdated.

github-actions bot commented Oct 19, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 19, 2023

Codecov Report

andrross left a comment

Choose a reason for hiding this comment

mch2 commented Oct 19, 2023 •

edited

Loading

github-actions bot commented Oct 19, 2023 •

edited

Loading