Failure of ConnectionRateLimitTest.connection_rate_test

[Lazin]

Assertion failure on CDT

FAIL test: ConnectionRateLimitTest.connection_rate_test (1/34 runs)
  failure at 2022-06-29T10:47:24.923Z: AssertionError()
      in job https://buildkite.com/redpanda/vtools/builds/2719#0181ae4a-637b-4455-a3d4-f3cf946742c9

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.9/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.9/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/tests/connection_rate_limit_test.py", line 122, in connection_rate_test
    assert time2 >= time1 * 1.7
AssertionError

https://buildkite.com/redpanda/vtools/builds/2719#0181ae4a-637b-4455-a3d4-f3cf946742c9/2952-3752

[ajfabbri]

Showed up in CI failures here, reopening.

[jcsp]

Here it is on dedicated EC2 instances where there shouldn't be a lot of background noise:
FAIL test: ConnectionRateLimitTest.connection_rate_test (1/2 runs)
failure at 2022-08-01T10:40:48.978Z: AssertionError('Time for first iteration:2.0255379676818848 Time for second iteration:3.049610662460327')
in job https://buildkite.com/redpanda/vtools/builds/3077#01825803-32b3-4fee-8e12-243a0177833c

[mmaslankaprv]

Another instance: https://buildkite.com/redpanda/redpanda/builds/13393#01825910-229a-4d3c-a843-9c15917ac087

[dotnwat]

Looks like it's again here https://buildkite.com/redpanda/vtools/builds/3139#01827b68-ebc9-4ff8-867f-d9461479bafe on CDT

[abhijat]

seen in https://buildkite.com/redpanda/redpanda/builds/13854#0182827a-dc8f-4cea-b367-4ef3aad5b793 on #5915

test_id:    rptest.tests.connection_rate_limit_test.ConnectionRateLimitTest.connection_rate_test
--
  | status:     FAIL
  | run time:   1 minute 51.757 seconds
  |  
  |  
  | AssertionError('Time for first iteration:2.1174551486968993 Time for second iteration:3.314431619644165')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/tests/connection_rate_limit_test.py", line 129, in connection_rate_test
  | assert time2 >= time1 * 1.7, f'Time for first iteration:{time1} Time for second iteration:{time2}'
  | AssertionError: Time for first iteration:2.1174551486968993 Time for second iteration:3.314431619644165

[BenPope]

https://ci-artifacts.dev.vectorized.cloud/redpanda/018280fd-2664-4b07-b855-4106cfb15c7b/vbuild/ducktape/results/2022-08-09--001/ConnectionRateLimitTest/connection_rate_test/19/

[abhijat]

seen on #5934

https://buildkite.com/redpanda/redpanda/builds/13929#0182885d-f1e9-4264-adb7-a29b802dba0c

test_id:    rptest.tests.connection_rate_limit_test.ConnectionRateLimitTest.connection_rate_test
  | status:     FAIL
  | run time:   1 minute 51.456 seconds
  |  
  |  
  | AssertionError('Time for first iteration:2.011513924598694 Time for second iteration:3.411939859390259')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/tests/connection_rate_limit_test.py", line 129, in connection_rate_test
  | assert time2 >= time1 * 1.7, f'Time for first iteration:{time1} Time for second iteration:{time2}'
 

[BenPope]

https://buildkite.com/redpanda/redpanda/builds/13913#0182862a-0033-436f-b2cc-44a8dac715b7

[BenPope]

https://buildkite.com/redpanda/redpanda/builds/13961#01828b6b-db28-422b-95cc-2220485d9e17

[BenPope]

https://buildkite.com/redpanda/redpanda/builds/14050#0182907f-eba9-4ba5-9657-2ef4ccc8b45b

[VladLazar]

https://buildkite.com/redpanda/redpanda/builds/14056#01829136-bfdf-46c5-a75c-152ac4e2570d

[rystsov]

Another instance: https://buildkite.com/redpanda/redpanda/builds/14414#0182b7fe-50eb-493f-9fcc-5bd0e66b6612