[Remote Store] Add checks to skip remote uploads after shard is closed #13998

shourya035 · 2024-06-05T12:06:08Z

Description

Adding checks in RemoteStoreRefreshListener to return false for shouldSync if the underlying indexShard instance has transitioned to a CLOSED state. This would prevent the shouldRetry method to schedule refresh retries even after the IndexShard instance and it's corresponding Remote directories are closed.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

#13996

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
~~API changes companion pull request created.~~
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-06-05T13:13:36Z

❌ Gradle check result for 6358f71: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-05T14:24:20Z

❌ Gradle check result for 6358f71: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

github-actions · 2024-06-06T05:53:29Z

✅ Gradle check result for 6358f71: SUCCESS

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-06-06T07:12:28Z

❌ Gradle check result for ebdc32c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

shourya035 · 2024-06-06T07:37:54Z

org.opensearch.cache.common.tier.TieredSpilloverCacheStatsIT.testTierLevelAggregation

Reopened: #6124

org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

Opened: #14029

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2024-06-06T09:16:15Z

❌ Gradle check result for bf2e0e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

shourya035 · 2024-06-06T09:28:36Z

❌ Gradle check result for bf2e0e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

org.opensearch.indices.IndicesRequestCacheIT.testDeleteAndCreateSameIndexShardOnSameNode

#13600

org.opensearch.indices.IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices {p0={"search.concurrent_segment_search.enabled":"false"}}

#13437

github-actions · 2024-06-06T09:39:36Z

❌ Gradle check result for bf2e0e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-06T12:10:41Z

✅ Gradle check result for bf2e0e9: SUCCESS

codecov · 2024-06-06T12:11:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.58%. Comparing base (b15cb0c) to head (bf2e0e9).
Report is 353 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13998      +/-   ##
============================================
+ Coverage     71.42%   71.58%   +0.16%     
- Complexity    59978    61316    +1338     
============================================
  Files          4985     5065      +80     
  Lines        282275   288201    +5926     
  Branches      40946    41741     +795     
============================================
+ Hits         201603   206309    +4706     
- Misses        63999    64852     +853     
- Partials      16673    17040     +367

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

#13998) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> (cherry picked from commit b01b6e8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#13998) (#14035) (cherry picked from commit b01b6e8) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-project#13998) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

opensearch-project#13998) (opensearch-project#14035) (cherry picked from commit b01b6e8) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

opensearch-project#13998) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Adding checks to skip remote refresh retry after shard is closed

6358f71

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

shourya035 changed the title ~~Adding checks to skip remote refresh retry after shard is closed~~ [Remote Store] Adding checks to skip remote refresh retry after shard is closed Jun 5, 2024

shourya035 changed the title ~~[Remote Store] Adding checks to skip remote refresh retry after shard is closed~~ [Remote Store] Adding checks to skip remote uploads retry after shard is closed Jun 5, 2024

shourya035 self-assigned this Jun 5, 2024

shourya035 added the skip-changelog label Jun 5, 2024

shourya035 marked this pull request as ready for review June 5, 2024 12:38

shourya035 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners June 5, 2024 12:38

gbbafna reviewed Jun 6, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java Outdated Show resolved Hide resolved

ashking94 approved these changes Jun 6, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java Outdated Show resolved Hide resolved

server/src/test/java/org/opensearch/index/shard/RemoteStoreRefreshListenerTests.java Show resolved Hide resolved

Addressing comments

ebdc32c

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

shourya035 added the backport 2.x Backport to 2.x branch label Jun 6, 2024

Retrigger integs

bf2e0e9

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

gbbafna approved these changes Jun 6, 2024

View reviewed changes

gbbafna merged commit b01b6e8 into opensearch-project:main Jun 6, 2024
30 checks passed

opensearch-trigger-bot bot mentioned this pull request Jun 6, 2024

[Backport 2.x] [Remote Store] Add checks to skip remote uploads after shard is closed #14035

Merged

prudhvigodithi mentioned this pull request Jun 7, 2024

Add additional details on Gradle Check failures autocut issues #13950

Closed

parv0201 pushed a commit to parv0201/OpenSearch that referenced this pull request Jun 10, 2024

[Remote Store] Add checks to skip remote uploads after shard is closed (

adca083

opensearch-project#13998) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

finnegancarroll mentioned this pull request Jun 14, 2024

Remove composeBuild fixture tasks when docker support not found #14357

Merged

3 tasks

prudhvigodithi mentioned this pull request Jun 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

opensearch-ci-bot mentioned this pull request Jun 17, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request Aug 22, 2024

[Remote Store] Add checks to skip remote uploads after shard is closed (

d7b8b78

opensearch-project#13998) Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add checks to skip remote uploads after shard is closed #13998

[Remote Store] Add checks to skip remote uploads after shard is closed #13998

shourya035 commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

shourya035 commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

shourya035 commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

codecov bot commented Jun 6, 2024

[Remote Store] Add checks to skip remote uploads after shard is closed #13998

[Remote Store] Add checks to skip remote uploads after shard is closed #13998

Conversation

shourya035 commented Jun 5, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

github-actions bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

shourya035 commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

shourya035 commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

codecov bot commented Jun 6, 2024

Codecov Report

shourya035 commented Jun 5, 2024 •

edited

Loading