-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failure in MaintenanceTest.test_maintenance_sticky.use_rpk=True #4772
Comments
This is a TimeoutError but it's in a part of the test waiting for leadership, so conceivably the same issue: |
Hmm, this just failed twice on https://buildkite.com/redpanda/redpanda/builds/12297#0181dd0a-fbe1-4f73-915e-5e5e8dd3ad0c Although that particular PR shouldn't have made leadership balancer any more aggressive, it was about throttling it. |
Looking at a failure log, it seems like leader balancer is trying to move leaderships to a node in maintenance mode, which fails, those groups get muted, and then when the tests expects them to get migrated after maintenance mode is over, that doesn't happen because we're still in the mute period. Could be that the test used to work because leader balances weren't fast enough to all run through and trigger mutes right away. |
The real fix will be to make the leader balancer aware of maintenance mode, but the test has become much more unstable since recent leader balancer changes to do more movements concurrently, so for the moment just run the test with the leader balancer disabled. Related: redpanda-data#4772
The real fix will be to make the leader balancer aware of maintenance mode, but the test has become much more unstable since recent leader balancer changes to do more movements concurrently, so for the moment just run the test with the leader balancer disabled. Related: redpanda-data#4772
The real fix will be to make the leader balancer aware of maintenance mode, but the test has become much more unstable since recent leader balancer changes to do more movements concurrently, so its worth mitigating that. The workaround is to set a short mute timeout so that muting nodes has no real effect, and a short idle timeout so that post-maintenance leader movements happen promptly. Related: redpanda-data#4772
The real fix will be to make the leader balancer aware of maintenance mode, but the test has become much more unstable since recent leader balancer changes to do more movements concurrently, so its worth mitigating that. The workaround is to set a short mute timeout so that muting nodes has no real effect, and a short idle timeout so that post-maintenance leader movements happen promptly. Related: redpanda-data#4772
Build (v22.1.x): https://buildkite.com/redpanda/redpanda/builds/10184#2e1a4495-7ef5-4a77-9d1c-0d1002b71ed8
Error
The text was updated successfully, but these errors were encountered: