SSHException in CompactionEndToEndTest.test_basic_compaction

[BenPope]

https://buildkite.com/redpanda/vtools/builds/4911#01854d77-3803-4f06-94df-09d2713c4046
https://buildkite.com/redpanda/vtools/builds/4916#0185529d-3893-43cc-a82e-6524ae0839d8

[INFO  - 2022-12-26 10:07:04,275 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.compaction_end_to_end_test.CompactionEndToEndTest.test_basic_compaction.key_set_cardinality=10.initial_cleanup_policy=compact.transactions=True.tx_inject_aborts=True: Summary: Exception('VerifiableProducer-0-139743812620816-worker-1: Traceback (most recent call last):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/home/ubuntu/redpanda/tests/rptest/services/verifiable_producer.py", line 218, in _worker\n    for line in node.account.ssh_capture(cmd):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 652, in next\n    return next(self.iter_obj)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 318, in output_generator\n    for line in iter(stdout.readline, \'\'):\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/file.py", line 291, in readline\n    new_data = self._read(n)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/channel.py", line 1361, in _read\n    return self.channel.recv(size)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/channel.py", line 710, in recv\n    self.transport._send_user_message(m)\n  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1914, in _send_user_message\n    raise SSHException(\nparamiko.ssh_exception.SSHException: Key-exchange timed out waiting for key negotiation\n')
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/compaction_end_to_end_test.py", line 140, in test_basic_compaction
    self.producer.stop()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 86, in stop
    self._propagate_exceptions()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
    raise Exception(self.errors)
Exception: VerifiableProducer-0-139743812620816-worker-1: Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
    self._worker(idx, node)
  File "/home/ubuntu/redpanda/tests/rptest/services/verifiable_producer.py", line 218, in _worker
    for line in node.account.ssh_capture(cmd):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 652, in next
    return next(self.iter_obj)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 318, in output_generator
    for line in iter(stdout.readline, ''):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/file.py", line 291, in readline
    new_data = self._read(n)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/channel.py", line 1361, in _read
    return self.channel.recv(size)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/channel.py", line 710, in recv
    self.transport._send_user_message(m)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1914, in _send_user_message
    raise SSHException(
paramiko.ssh_exception.SSHException: Key-exchange timed out waiting for key negotiation


[INFO  - 2022-12-26 10:07:04,281 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.compaction_end_to_end_test.CompactionEndToEndTest.test_basic_compaction.key_set_cardinality=10.initial_cleanup_policy=compact.transactions=True.tx_inject_aborts=True: Data: None

Looks similar to #6792

[jcsp]

Same as #6792

We should fix for good this by changing VerifiableProducer to avoid using BackgroundThreadedService.

[piyushredpanda]

@bharathv: could you help out with the proper fix given the discussion in #6792 ?

[piyushredpanda]

Hi @rystsov, I think I saw in notes that you were helping take up the SSH servicification, so this probably belongs better with you?

[rystsov]

sure, I'll take care of it

[andijcr]

https://buildkite.com/redpanda/vtools/builds/5492#0185ed20-8fbc-492a-b902-3174d5125678

FAIL test: CompactionEndToEndTest.test_basic_compaction.key_set_cardinality=10.initial_cleanup_policy=delete.transactions=True.tx_inject_aborts=True (1/42 runs)
  failure at 2023-01-26T15:48:03.990Z: Exception('VerifiableProducer-0-140340962647568-worker-1: Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
    self._worker(idx, node)
  File "/home/ubuntu/redpanda/tests/rptest/services/verifiable_producer.py", line 218, in _worker
    for line in node.account.ssh_capture(cmd):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 652, in next
    return next(self.iter_obj)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 318, in output_generator
    for line in iter(stdout.readline, \'\'):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/file.py", line 291, in readline
    new_data = self._read(n)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/channel.py", line 1361, in _read
    return self.channel.recv(size)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/channel.py", line 710, in recv
    self.transport._send_user_message(m)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/paramiko/transport.py", line 1914, in _send_user_message
    raise SSHException(
paramiko.ssh_exception.SSHException: Key-exchange timed out waiting for key negotiation
')
      on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/5492#0185ed20-8fbc-492a-b902-3174d5125678

not really related to the test itself

[andijcr]

related: #8458

[jcsp]

This should be covered by the ducktape update that sets paramiko.packet.Packetizer.REKEY_BYTES
https://github.com/redpanda-data/vtools/pull/1420