Failure in MaintenanceTest.test_maintenance_sticky.use_rpk=True

[dimitriscruz]

Build (v22.1.x): https://buildkite.com/redpanda/redpanda/builds/10184#2e1a4495-7ef5-4a77-9d1c-0d1002b71ed8

FAIL test: MaintenanceTest.test_maintenance_sticky.use_rpk=True (1/3 runs)
  failure at 2022-05-16T14:38:18.499Z: AssertionError('`rpk cluster maintenance status` has changed: [\'Request\', \'error,\', \'trying\', \'another\', \'node:\', \'request\', \'failed:\', \'Service\', \'Unavailable,\', \'body:\', \'"{\\\\"message\\\\":\', \'\\\\"Unable\', \'to\', \'get\', \'cluster\', \'health:\', \'Currently\', \'there\', \'is\', \'no\', \'leader\', \'controller\', \'elected\', \'in\', \'the\', \'cluster\\\\",\', \'\\\\"code\\\\":\', \'503}"\']')
      in job https://buildkite.com/redpanda/redpanda/builds/10184#2e1a4495-7ef5-4a77-9d1c-0d1002b71ed8

Error

test_id:    rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=True
--
  | status:     FAIL
  | run time:   9 minutes 1.238 seconds
  |  
  |  
  | AssertionError('`rpk cluster maintenance status` has changed: [\'Request\', \'error,\', \'trying\', \'another\', \'node:\', \'request\', \'failed:\', \'Service\', \'Unavailable,\', \'body:\', \'"{\\\\"message\\\\":\', \'\\\\"Unable\', \'to\', \'get\', \'cluster\', \'health:\', \'Currently\', \'there\', \'is\', \'no\', \'leader\', \'controller\', \'elected\', \'in\', \'the\', \'cluster\\\\",\', \'\\\\"code\\\\":\', \'503}"\']')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/maintenance_test.py", line 225, in test_maintenance_sticky
  | self._verify_cluster(None, False)
  | File "/root/tests/rptest/tests/maintenance_test.py", line 175, in _verify_cluster
  | wait_until(
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 53, in wait_until
  | raise e
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 44, in wait_until
  | if condition():
  | File "/root/tests/rptest/tests/maintenance_test.py", line 176, in <lambda>
  | lambda: self._verify_maintenance_status(node, expect),
  | File "/root/tests/rptest/tests/maintenance_test.py", line 87, in _verify_maintenance_status
  | statuses = self.rpk.cluster_maintenance_status()
  | File "/root/tests/rptest/clients/rpk.py", line 553, in cluster_maintenance_status
  | return list(filter(None, map(parse, output.splitlines())))
  | File "/root/tests/rptest/clients/rpk.py", line 528, in parse
  | assert len(
  | AssertionError: `rpk cluster maintenance status` has changed: ['Request', 'error,', 'trying', 'another', 'node:', 'request', 'failed:', 'Service', 'Unavailable,', 'body:', '"{\\"message\\":', '\\"Unable', 'to', 'get', 'cluster', 'health:', 'Currently', 'there', 'is', 'no', 'leader', 'controller', 'elected', 'in', 'the', 'cluster\\",', '\\"code\\":', '503}"']
  |  

[jcsp]

This is a TimeoutError but it's in a part of the test waiting for leadership, so conceivably the same issue:
https://buildkite.com/redpanda/redpanda/builds/12286#0181dc4d-53e3-40ee-b939-2a2e58085864

[jcsp]

Hmm, this just failed twice on dev right after I merged #5159, so maybe that wasn't a coincidence.

https://buildkite.com/redpanda/redpanda/builds/12297#0181dd0a-fbe1-4f73-915e-5e5e8dd3ad0c

Although that particular PR shouldn't have made leadership balancer any more aggressive, it was about throttling it.

[jcsp]

Looking at a failure log, it seems like leader balancer is trying to move leaderships to a node in maintenance mode, which fails, those groups get muted, and then when the tests expects them to get migrated after maintenance mode is over, that doesn't happen because we're still in the mute period.

Could be that the test used to work because leader balances weren't fast enough to all run through and trigger mutes right away.

[LenaAn]

another one: https://buildkite.com/redpanda/redpanda/builds/12722#01821639-c833-4786-b045-29a0e1a29f5b from #5484

[twmb]

https://buildkite.com/redpanda/redpanda/builds/12767#01821925-7ca4-4225-9088-7971d1272679/1583-8860

[twmb]

https://buildkite.com/redpanda/redpanda/builds/12758#018218b5-afd6-4e91-af39-ada13e1c3ddf/1501-9102

[twmb]

https://buildkite.com/redpanda/redpanda/builds/12758#018218b5-afd6-4e91-af39-ada13e1c3ddf/1501-9102

[LenaAn]

another one from #5484
https://buildkite.com/redpanda/redpanda/builds/12850#018220d1-a930-4179-99b9-f43e1612c9d3

[rystsov]

another occurrence: https://buildkite.com/redpanda/redpanda/builds/12899#0182246f-263f-4db9-ba6a-0b2d60eeeb8c