CI Failure (failed to stop after SIGKILL) in ConsumerGroupTest.test_dead_group_recovery

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/19571#0184f582-22ce-49b9-80cb-5c237459b23f

Module: rptest.tests.consumer_group_test
Class:  ConsumerGroupTest
Method: test_dead_group_recovery
Arguments:
{
  "static_members": true
}

[INFO  - 2022-12-09 06:51:27,855 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.consumer_group_test.ConsumerGroupTest.test_dead_group_recovery.static_members=True: Summary: TimeoutError('cli-consumer-0 running on docker-rp-30 failed to stop after SIGKILL')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/consumer_group_test.py", line 357, in test_dead_group_recovery
    c.stop()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 84, in stop
    super(BackgroundThreadService, self).stop()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/service.py", line 284, in stop
    self.stop_node(node, **kwargs)
  File "/root/tests/rptest/services/kafka_cli_consumer.py", line 117, in stop_node
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: cli-consumer-0 running on docker-rp-30 failed to stop after SIGKILL

[INFO  - 2022-12-09 06:51:27,858 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.consumer_group_test.ConsumerGroupTest.test_dead_group_recovery.static_members=True: Data: None

[piyushredpanda]

Do we have the PR for this now, @rystsov (the service-ification of the test to bypass SSH errors)?

[dotnwat]

sev/low: because if the kafka command line tools don't exit on sigkill then it's either a problem with upstream kafka cli tooling (or the version we are using) or something wrong with ducktape failing to properly deliver the signal to the container to shutdown.

closing: no reports of this in over 2 months.