[BUG] Flaky test in permissions_spec.js in functional-tests dashboards repo

[DarshitChanpura]

Intermittent test failure for a test in permission_spec.js with unknown root cause.

More specifically cy.wait(@getPermissions) gets stuck on waiting for mock command to finish execution.

Failed once during 2.2.0 build and failed recently during 2.2.1 build.

How can one reproduce the bug?
It is difficult because of the flakiness but you can run this command to run the tests in the file:

yarn cypress:run-plugin-tests-with-security --spec "cypress/integration/plugins/security/permissions_spec.js"

What is the expected behavior?
Tests should not have intermittent failures.

Do you have any screenshots?
security.zip

[DarshitChanpura]

This seems related: cypress-io/cypress#18524

[peternied]

This seems like its an intermittent failure - sounds like cypress isn't working correctly for this scenario.

[RyanL1997]

We just confirmed that this issue is still existing in the current release of 2.5.0. I have re-run manually on my local machine, and it seems like it only gets unstable during the running of workflows.

[peternied]

@RyanL1997 Could you include detail on how you fixed the flakiness and how you confirmed it won't reproduce again?

This could look like;

Found a timing issues where an multiple asynchronous actions X and Y were racing #561 - Confirmed this by running the test 10 times in GitHub action on my fork and saw zero failures, [link to test run report].

[RyanL1997]

@RyanL1997 Could you include detail on how you fixed the flakiness and how you confirmed it won't reproduce again?

This could look like;

Found a timing issues where an multiple asynchronous actions X and Y were racing #561 - Confirmed this by running the test 10 times in GitHub action on my fork and saw zero failures, [link to test run report].

I have run this test 30 times in a row on my local machine and none of them failed. However, since the original error is hard to reproduce it, it is hard to compare the failing rate to this fix. But I do have a run in my own fork (Like 8 times?), and none of them failed by the original flakiness. It only failed couple times for the windows runner connections issues of github:

https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/4230492547

Here is the experiment I did. I was trying to run it 10 times in a row, but it got cancelled at 9th time, cuz I was doing a rebase, and it discarded the changes on main.

[peternied]

Thanks - just in case this issue gets re-opened we've got some context, flaky tests are difficult to deal with at the best of times

[RyanL1997]

Thanks - just in case this issue gets re-opened we've got some context, flaky tests are difficult to deal with at the best of times

Ya, 100% agree. I will keep tracking on this.