Failure in consumer_offsets_migration_test.ConsumerOffsetsMigrationTest.test_migrating_consume_offsets.failures=True

[gousteris]

bk ref:

• https://buildkite.com/redpanda/redpanda/builds/8235#6d15c3e8-ae9d-49f9-b938-8f7c28dce246/1525-6994
• https://buildkite.com/redpanda/redpanda/builds/8243#d3cf4e72-f60e-4b60-8e9c-1e0e954cd57b/1563-7583

test_id:    rptest.tests.consumer_offsets_migration_test.ConsumerOffsetsMigrationTest.test_migrating_consume_offsets.failures=True
--
  | status:     FAIL
  | run time:   1 minute 17.785 seconds
  |  
  |  
  | NodeNotReadyError()
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 215, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/consumer_offsets_migration_test.py", line 130, in test_migrating_consume_offsets
  | wait_until(_consumer_offsets_present, timeout_sec=90, backoff_sec=3)
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 53, in wait_until
  | raise e
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 44, in wait_until
  | if condition():
  | File "/root/tests/rptest/tests/consumer_offsets_migration_test.py", line 127, in _consumer_offsets_present
  | self.client().describe_topic("__consumer_offsets"))
  | File "/root/tests/rptest/clients/default.py", line 101, in describe_topic
  | td = self.describe_topics([topic])
  | File "/root/tests/rptest/clients/default.py", line 94, in describe_topics
  | client = KafkaAdminClient(
  | File "/usr/local/lib/python3.9/dist-packages/kafka/admin/client.py", line 218, in __init__
  | self._refresh_controller_id()
  | File "/usr/local/lib/python3.9/dist-packages/kafka/admin/client.py", line 278, in _refresh_controller_id
  | controller_version = self._client.check_version(controller_id, timeout=(self.config['api_version_auto_timeout_ms'] / 1000))
  | File "/usr/local/lib/python3.9/dist-packages/kafka/client_async.py", line 909, in check_version
  | version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
  | File "/usr/local/lib/python3.9/dist-packages/kafka/conn.py", line 1240, in check_version
  | raise Errors.NodeNotReadyError()
  | kafka.errors.NodeNotReadyError: NodeNotReadyError
  |  

[dotnwat]

Another one here

https://buildkite.com/redpanda/redpanda/builds/8305#8d26a372-e49c-4c12-b8f2-af601eb184c4

#4058

[Lazin]

Another one here - https://buildkite.com/redpanda/redpanda/builds/8321#ed65bed9-0cc0-4341-a4e5-49b0188f2824/1665-7470

#3195

[andrewhsu]

another one here in this PR that did not have code changes: https://buildkite.com/redpanda/redpanda/builds/8351#85afac3d-7268-42a0-b035-e50bcda328c4/1530-7368

#4088

[VadimPlh]

Another one https://buildkite.com/redpanda/redpanda/builds/8378#773ff86b-7a5f-4c48-bc7a-58aec4b06a42

[mmaslankaprv]

working on it

[andrewhsu]

/backport v21.11.x

[mmaslankaprv]

This doesn't need a backport to v21.11.x, consumer offset feature is not present in v21 versions.

[andrewhsu]

seen again: https://buildkite.com/redpanda/redpanda/builds/8517#10a417de-1b86-4f50-823f-0099de3e2bb2/1546-8983

[gousteris]

seen here https://buildkite.com/redpanda/redpanda/builds/8703 and seems that it failed after closing this issue @mmaslankaprv I might be confused can you double check please?