[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreClusterStateRestoreIT

[opensearch-ci-bot]

Flaky Test Report for RemoteStoreClusterStateRestoreIT

Noticed the RemoteStoreClusterStateRestoreIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
ccf5289	14221	40884	org.opensearch.remotestore.RemoteStoreClusterStateRestoreIT.testFullClusterRestoreGlobalMetadata

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the RemoteStoreClusterStateRestoreIT class are:

• 13924
• 13637
• 13865
• 14264
• 14338
• 14345

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[shiv0408]

This test should not be flaky now, the fix was merged in #14230

[shiv0408]

@prudhvigodithi Wanted to confirm if a test becomes flaky again after the issue has been resolved, does our workflow create a new issue or open the previous closed issue?

[prudhvigodithi]

Hey @shiv0408 yes the automation will re-open the issue if it was closed within 3 days (which is configurable) else it will re-create a new issue. In both cases the issue body will have the latest flaky test information.
Thanks
@getsaurabh02

[shiv0408]

@andrross Do you know how this issue got re-opened? I closed the issue as the fix was made, do we need to keep the issue open?

[andrross]

@shiv0408 This is the PR that caused it to reopen: #14345

I think @prudhvigodithi is looking into a similar case. It might be that a PR is open that hasn't rebased with your fix, and that causes it to reopen if the test fails.

[shiv0408]

This PR was merged 3 days ago, but the CI bot opened is couple of hours ago.

[prudhvigodithi]

The automation looks for last 30 days build data in post merge action and if found RemoteStoreClusterStateRestoreIT will re-open/create the issue. So even though the fix was pushed, the RemoteStoreClusterStateRestoreIT was found failing in past 30 days hence it re-opened the existing issue.

How about we allow the automation to close the issue? Since now we have all the issues created for the flaky tests, we can reduce to identify the flaky tests in last 15 days and auto-close the issue if not failing in last 15 days, this way the user need not worry about closing the issue ? WDYT @andrross @shiv0408
Adding @dblock @reta @getsaurabh02

[shiv0408]

How about we allow the automation to close the issue?

This might present incorrect information that test is still flaky, while it might have been resolved. Anyway, we are going to reopen the issue if test turns to be flaky again.

[andrross]

@prudhvigodithi If an issue exists and it is closed, and none of the failures are newer than the close date, then can we keep it closed?

[prudhvigodithi]

Sure, what we can do it the following:

• The automation will go back 30 days and check for failures in post merge action, flag the tests and create a detailed issue report. The issue body will be updated with the latest information as and when it has the latest information which will be indexed through the Gradle Check build. This is the current state.

• If the issue is closed (considering the flaky test is fixed by the user) the automation should not re-open unless the data is different from what shown in the issue body, if anything (the issue body) is different after closed then it should re-open the issue. Here the data to compare is the markdown table and not the linked PR's as during the PR creation the failures sometimes could be genuine. So re-open when seen a new failure (with a different post merge commit) after the issue is closed. This should also solve the problem where sometimes we think the Flaky test is fixed but would re-occur and with new reoccurrence the issue should re-open with new data.

[peterzhuamazon]

Can we compare the base of the PRs to determine if such issue is legit or we inform user to rebase?

It is possible that an issue can re-surface due to regression.

At least if we see the PR has a updated base, we can re-open the old issue if needed. Else, inform user to rebase their branch. Thanks.

[shiv0408]

@prudhvigodithi we should create an issue to add your second point as an enhancement on our current state.

[dblock]

[Catch All Triage - Attendees 1, 2, 3, 4, 5]