-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Timeout failure in EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures
#5390
Comments
this is similar to #4639 |
The last example of this failure mode was 3 weeks ago: More recent failures of this test have all been #4639 |
That last one is a "failed to consume up to", so #4639 |
I've figured this specific failure mode out ( The test waits for the number of segments to become lesser or equal to 6 while randomly killing nodes.
For each nod being killed, the current segment is rolled and a new one is created. There's a number of things we can do in order to address this issue with the test:
I could use a second opinion here. @LenaAn, @jcsp what do you think? |
@Lazin and @abhijat as FYIs above. Amazing sleuthing, @VladLazar! |
Great find!
FWIW I tried to increase min time between failures in a test branch to 20 seconds IIRC and the failure became much rarer but was still present in 1 of 500 runs in my local setup. Your third option seems more viable to me specifically for this test. |
Option 3 please -- as you say, the test is wrong to assert on the segment count in the presence of failures.
|
https://buildkite.com/redpanda/redpanda/builds/12243#0181d907-c5e0-4ccd-ad4e-87f624f9ae5d/1595-8717
The text was updated successfully, but these errors were encountered: