Perform unreferenced file cleanup for any operation failure and mute flaky test #12128

RS146BIJAY · 2024-02-01T13:34:23Z

Description

As of now, we cleanup unreferenced files whenever last write is performed by merge and it caused disk to get full and shard to fail. Incase some other operation performs the last write and caused disk to get 100% full and a merge is ongoing, merge will just get aborted and no cleanup will be performed. Since when closing the shard it is the other operation causing disk full and not segment merge.

In order to fix this we need to cleanup unreferenced files when any operation failed due to disk full.

Related Issues

#12054

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-02-01T13:53:53Z

Compatibility status:

Checks if related components are compatible with change 56424ec

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/alerting.git]

github-actions · 2024-02-01T14:24:15Z

❌ Gradle check result for 22ba8eb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…flaky tests Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com>

github-actions · 2024-02-01T16:32:27Z

✅ Gradle check result for eb68176: SUCCESS

codecov · 2024-02-01T16:34:13Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.34%. Comparing base (4471a8d) to head (56424ec).
Report is 378 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12128      +/-   ##
============================================
+ Coverage     71.30%   71.34%   +0.04%     
- Complexity    59393    59507     +114     
============================================
  Files          4925     4925              
  Lines        279540   279539       -1     
  Branches      40646    40645       -1     
============================================
+ Hits         199333   199448     +115     
+ Misses        63580    63517      -63     
+ Partials      16627    16574      -53

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: rishavz_sagar <rishavsagar4b1@gmail.com>

github-actions · 2024-02-01T17:44:38Z

❕ Gradle check result for 56424ec: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

deshsidd · 2024-02-01T19:11:32Z

server/src/main/java/org/opensearch/index/engine/Engine.java

+                    // clean up all unreferenced files on best effort basis created during failed merge and reset the
+                    // shard state back to last Lucene Commit
+                    if (shouldCleanupUnreferencedFiles() && isOperationFailureDueToIOException(failure)) {
+                        logger.info("Cleaning up unreferenced files created during failed merge due to: {}", reason);


Should be warning?

We would be doing extra clean ups on may be other shards as well due to this . Please evaluate if we can avoid the same and do clean up only on the responsible shard instead

opensearch-trigger-bot · 2024-03-05T15:19:45Z

This PR is stalled because it has been open for 30 days with no activity.

Bukhtawar · 2024-03-25T17:22:57Z

server/src/main/java/org/opensearch/index/engine/Engine.java

+                    if (shouldCleanupUnreferencedFiles() && isOperationFailureDueToIOException(failure)) {
+                        logger.info("Cleaning up unreferenced files created during failed merge due to: {}", reason);


Do we need to log if the exception isn't IOE

gbbafna · 2024-04-04T15:09:18Z

@RS146BIJAY : are you still working on this change ?

opensearch-trigger-bot · 2024-05-07T15:20:24Z

This PR is stalled because it has been open for 30 days with no activity.

Perform unreferenced file cleanup for any operation failure and mute …

eb68176

…flaky tests Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com>

RS146BIJAY force-pushed the seg_merge branch from 22ba8eb to eb68176 Compare February 1, 2024 15:46

Merge branch 'main' into seg_merge

56424ec

Signed-off-by: rishavz_sagar <rishavsagar4b1@gmail.com>

RS146BIJAY marked this pull request as ready for review February 1, 2024 17:54

RS146BIJAY requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, ryanbogan, sachinpkale and saratvemulapalli as code owners February 1, 2024 17:54

RS146BIJAY requested review from shwetathareja, sohami, tlfeng and VachaShah as code owners February 1, 2024 17:54

RS146BIJAY mentioned this pull request Feb 1, 2024

[BUG] Cleanup unreferenced files when any operation failed due to disk full. #12054

Open

deshsidd reviewed Feb 1, 2024

View reviewed changes

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Mar 5, 2024

Bukhtawar reviewed Mar 25, 2024

View reviewed changes

gbbafna added the stalled Issues that have stalled label Apr 4, 2024

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Apr 4, 2024

opensearch-trigger-bot bot added the stalled Issues that have stalled label May 7, 2024

RS146BIJAY closed this May 8, 2024

RS146BIJAY deleted the seg_merge branch July 8, 2024 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perform unreferenced file cleanup for any operation failure and mute flaky test #12128

Perform unreferenced file cleanup for any operation failure and mute flaky test #12128

RS146BIJAY commented Feb 1, 2024 •

edited

Loading

github-actions bot commented Feb 1, 2024 •

edited

Loading

github-actions bot commented Feb 1, 2024

github-actions bot commented Feb 1, 2024

codecov bot commented Feb 1, 2024 •

edited

Loading

github-actions bot commented Feb 1, 2024

deshsidd Feb 1, 2024

gbbafna Feb 2, 2024

opensearch-trigger-bot bot commented Mar 5, 2024

Bukhtawar Mar 25, 2024

gbbafna commented Apr 4, 2024

opensearch-trigger-bot bot commented May 7, 2024

		if (shouldCleanupUnreferencedFiles() && isOperationFailureDueToIOException(failure)) {
		logger.info("Cleaning up unreferenced files created during failed merge due to: {}", reason);

Perform unreferenced file cleanup for any operation failure and mute flaky test #12128

Perform unreferenced file cleanup for any operation failure and mute flaky test #12128

Conversation

RS146BIJAY commented Feb 1, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Feb 1, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Feb 1, 2024

github-actions bot commented Feb 1, 2024

codecov bot commented Feb 1, 2024 • edited Loading

Codecov Report

github-actions bot commented Feb 1, 2024

deshsidd Feb 1, 2024

Choose a reason for hiding this comment

gbbafna Feb 2, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Mar 5, 2024

Bukhtawar Mar 25, 2024

Choose a reason for hiding this comment

gbbafna commented Apr 4, 2024

opensearch-trigger-bot bot commented May 7, 2024

RS146BIJAY commented Feb 1, 2024 •

edited

Loading

github-actions bot commented Feb 1, 2024 •

edited

Loading

codecov bot commented Feb 1, 2024 •

edited

Loading