test: test_maintenance_sticky: unable find controller leader

[dotnwat]

*https://buildkite.com/redpanda/redpanda/builds/9727#e03f53ba-c231-44b4-a2e9-6b62faaedbfe
*#4517

Appears to be an instance related to #3615 where on ARM we are encountering more and more delays and this delay is resulting in a leadership election for the controller that doesn't complete within the period of time that that rpk will retry on 503.

test_id:    rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=True
--
  | status:     FAIL
  | run time:   8 minutes 34.030 seconds
  |  
  |  
  | RpkException('command /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-05a93799c220159ae-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/bin/rpk --api-urls docker-rp-4:9644,docker-rp-31:9644,docker-rp-35:9644 cluster maintenance status returned 1, output: Request error, trying another node: request   failed: Service Unavailable, body: "{\\"message\\": \\"Unable to get cluster health: Currently there is no leader controller elected in the cluster\\", \\"code\\": 503}"\nRequest error, trying another node: request   failed: Service Unavailable, body: "{\\"message\\": \\"Unable to get cluster health: Currently there is no leader controller elected in the cluster\\", \\"code\\": 503}"\n', 'unable to request brokers: request   failed: Service Unavailable, body: "{\\"message\\": \\"Unable to get cluster health: Currently there is no leader controller elected in the cluster\\", \\"code\\": 503}"\n')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/maintenance_test.py", line 225, in test_maintenance_sticky
  | self._verify_cluster(None, False)
  | File "/root/tests/rptest/tests/maintenance_test.py", line 175, in _verify_cluster
  | wait_until(
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 53, in wait_until
  | raise e
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 44, in wait_until
  | if condition():
  | File "/root/tests/rptest/tests/maintenance_test.py", line 176, in <lambda>
  | lambda: self._verify_maintenance_status(node, expect),
  | File "/root/tests/rptest/tests/maintenance_test.py", line 87, in _verify_maintenance_status
  | statuses = self.rpk.cluster_maintenance_status()
  | File "/root/tests/rptest/clients/rpk.py", line 552, in cluster_maintenance_status
  | output = self._execute(cmd)
  | File "/root/tests/rptest/clients/rpk.py", line 488, in _execute
  | raise RpkException(
  | rptest.clients.rpk.RpkException: RpkException<command /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-05a93799c220159ae-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/bin/rpk --api-urls docker-rp-4:9644,docker-rp-31:9644,docker-rp-35:9644 cluster maintenance status returned 1, output: Request error, trying another node: request   failed: Service Unavailable, body: "{\"message\": \"Unable to get cluster health: Currently there is no leader controller elected in the cluster\", \"code\": 503}"
  | Request error, trying another node: request   failed: Service Unavailable, body: "{\"message\": \"Unable to get cluster health: Currently there is no leader controller elected in the cluster\", \"code\": 503}"
  | error: unable to request brokers: request   failed: Service Unavailable, body: "{\"message\": \"Unable to get cluster health: Currently there is no leader controller elected in the cluster\", \"code\": 503}"
  | >

[piyushredpanda]

Requesting if @rystsov could look into this given there was a similar ARM issue he had debugged recently, IIRC.

[jcsp]

Looks to be not ARM specific, seen on amd64 here:
https://buildkite.com/redpanda/redpanda/builds/9892#742a766b-0702-47ef-beb4-64caf3e9f317