No such process error in ConsumerOffsetsMigrationTest.test_migrating_consume_offsets.failures=True.cpus=1

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/10970#018137b8-8fb1-4bb2-8b85-736f7e7103af

Module: rptest.tests.consumer_offsets_migration_test
Class:  ConsumerOffsetsMigrationTest
Method: test_migrating_consume_offsets
Arguments:
{
  "cpus": 1,
  "failures": true
}

[INFO  - 2022-06-06 07:12:58,434 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.consumer_offsets_migration_test.ConsumerOffsetsMigrationTest.test_migrating_consume_offsets.failures=True.cpus=1: FAI
L: RemoteCommandError({'ssh_config': {'host': 'docker-rp-4', 'hostname': 'docker-rp-4', 'user': 'root', 'port': 22, 'password': 'UNUSED', 'identityfile': '/root/.ssh/id_rsa'}, 'hostname': 'docker-rp-4', 'ssh_hostna
me': 'docker-rp-4', 'user': 'root', 'externally_routable_ip': 'docker-rp-4', '_logger': <Logger rptest.tests.consumer_offsets_migration_test.ConsumerOffsetsMigrationTest.test_migrating_consume_offsets.failures=True
.cpus=1-40 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f015c310f70>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f0158d1a130>}, 'kill -15 3404', 1, b'bash: line
 1: kill: (3404) - No such process\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/consumer_offsets_migration_test.py", line 141, in test_migrating_consume_offsets
    self.redpanda.restart_nodes(n, stop_timeout=60)
  File "/root/tests/rptest/services/redpanda.py", line 1283, in restart_nodes
    self.stop_node(node, timeout=stop_timeout)
  File "/root/tests/rptest/services/redpanda.py", line 1117, in stop_node
    node.account.signal(pid, signal.SIGTERM, allow_fail=False)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 391, in signal
    self.ssh(cmd, allow_fail=allow_fail)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 275, in ssh
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@docker-rp-4: Command 'kill -15 3404' returned non-zero exit status 1. Remote error message: b'bash: line 1: kill: (3404) - No such process\n'

Looks like failure injector restarted a node (and its pid changed), but then the test tried to use the old pid to restart.

[bharathv]

Fixed in #5074.