-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failure in ControllerUpgradeTest
test_updating_cluster_when_executing_operations
#5378
Comments
happened just now, my branch is rebased to fresh dev https://buildkite.com/redpanda/redpanda/builds/13246#0182447c-878b-49aa-968d-7643cdab143b |
Before, the test failed to complete some number of operations within the timeout. Looks like we're hitting a new error now:
I'll dig in. |
Again Error is
|
seen on #5757 https://buildkite.com/redpanda/redpanda/builds/13415#018259ea-dff0-4d4e-8c72-89c30874ce5f test_id: rptest.tests.controller_upgrade_test.ControllerUpgradeTest.test_updating_cluster_when_executing_operations
Traceback (most recent call last): #5742 should fix this as it will downgrade the error to debug |
Seen in #5779
|
This is still open until #5742 |
Saw this again in the vtools build: Will put up a patch to catch the log, since this is an upgrade test. |
The AdminOperationsFuzzer retries operations 5 times before giving up. In this instance it is retrying a configuration update 5 times. The fuzzer starts trying just at the same time as the first node in the cluster is restarted, and retries every 1 second for 5 seconds. During this period the updates continue to fail, which is a little unexpected (one would expect fresh controller leader election to be quicker than this), but not inconceivable given the slow/noisy runtime environment.
I think this is a case where the AdminOperationsFuzzer needs a higher number of retries (or a longer delay between them): it is not really a failure for the cluster to go 5 seconds without being able to service a controller write, in this environment.
https://buildkite.com/redpanda/redpanda/builds/12215#0181d6c3-6183-4b8d-8897-60c2ad205a02
The text was updated successfully, but these errors were encountered: