Fix flaky RemoteStoreRefreshListenerTests testTrackerData #11256 #11803
Conversation
github-actions
bot
added
bug
ashking94
requested review from
abbashus,
adnapibar,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
dreamer-89,
gbbafna,
kartg,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
ryanbogan,
sachinpkale,
saratvemulapalli,
setiah,
shwetathareja,
sohami,
tlfeng and
VachaShah
as code owners
|
@opensearch-project/opensearch-core Please review the fix for flaky test. It is just a test fix. Since the refresh happens in background. This particular case was happening due to refresh not yet successful while the assertion kicked in. I am running this test with fix on loop overnight. I will confirm once the test has passed for over 10K iterations and then you merge this PR. |
Compatibility status:Checks if related components are compatible with change 16cec03 Incompatible componentsIncompatible components: [https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git] |
|
❕ Gradle check result for 4aa121d: UNSTABLE
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #11803 +/- ##
============================================
+ Coverage 71.33% 71.35% +0.01%
+ Complexity 59300 59271 -29
============================================
Files 4921 4911 -10
Lines 278989 278667 -322
Branches 40543 40528 -15
============================================
- Hits 199014 198836 -178
+ Misses 63444 63310 -134
+ Partials 16531 16521 -10 ☔ View full report in Codecov by Sentry. |
|
Saw the test failing at 965th iteration with below assertion failure stack trace - The reason for this is due to exception while uploading metadata for the refresh - |
|
❌ Gradle check result for 73fac30: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
org.opensearch.qa.verify_version_constants.VerifyVersionConstantsIT.testLuceneVersionConstant |
Signed-off-by: Ashish Singh <ssashish@amazon.com>
|
❌ Gradle check result for 16cec03: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Flaky tests - |
ashking94
changed the title
|
Have run the test for 2K+ iterations and no failures have occurred so far. |
|
❕ Gradle check result for 16cec03: UNSTABLE
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Flaky test - #9191 |
… (opensearch-project#11803) Signed-off-by: Ashish Singh <ssashish@amazon.com>
(cherry picked from commit 517f091) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
… (opensearch-project#11803) Signed-off-by: Ashish Singh <ssashish@amazon.com>
… (opensearch-project#11803) Signed-off-by: Ashish Singh <ssashish@amazon.com>
… (opensearch-project#11803) Signed-off-by: Ashish Singh <ssashish@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Description
This closes #11256. The flakiness in the
RemoteStoreRefreshListenerTests.testTrackerDatatest is happening due toRemoteSegmentStoreDirectory.init()method being called on account of simultaneous refreshes from test. In the test setup, we create a full fledged IndexShard and use it's remote store classes for mimicing/mocking certain test behaviours. Now, the refreshes of the underlying IndexShard and the manually invoked test's afterRefreshes were happening simultaneously causing the sharedRemoteSegmentStoreDirectoryobject'ssegmentsUploadedToRemoteStoremap to get cleared. The changes now ensures that the concurrency does not happen any more.Related Issues
Resolves #11256
Check List
New functionality has been documented.New functionality has javadoc addedCommit changes are listed out in CHANGELOG.md file (See: Changelog)Public documentation issue/PR createdBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.