Timeout in ManyPartitionsTest (ManyPartitionsTest.test_many_partitions)

[jcsp]

This is timing out trying to produce data, where the test expects to be able to achieve an average rate of 10MB/s
https://buildkite.com/redpanda/redpanda/builds/9213#4986e508-a6ea-4f58-8497-0c20ee249cf4

I'm going to check the log to make sure the cluster is really progressing, but this is probably a case where we just need to lower our expectations a bit further for the performance available in the docker test environment, and earmark this test for running on nightly EC2 runs instead of docker runs.

[jcsp]

The last produce messages that each node sees are

TRACE 2022-04-20 17:24:06,908 [shard 1] kafka - produce.cc:525 - handling produce request {transactional_id={nullopt} acks=-1 timeout_ms=5000 topics={{name={scale_000000} partitions={{partition_index=676 records={batch 131160 v2_format true valid_crc true}}}}}}
TRACE 2022-04-20 17:24:06,908 [shard 0] kafka - produce.cc:525 - handling produce request {transactional_id={nullopt} acks=-1 timeout_ms=5000 topics={{name={scale_000000} partitions={{partition_index=933 records={batch 131160 v2_format true valid_crc true}}}}}}
TRACE 2022-04-20 17:23:57,778 [shard 1] kafka - produce.cc:525 - handling produce request {transactional_id={nullopt} acks=-1 timeout_ms=5000 topics={{name={scale_000000} partitions={{partition_index=707 records={batch 131160 v2_format true valid_crc true}}, {partition_index=110 records={batch 131160 v2_format true valid_crc true}}, {partition_index=462 records={batch 131160 v2_format true valid_crc true}}}}}}

But the timeout is all the way forward at 17:27:04,555. So this isn't a simple slowness thing.

[jcsp]

In the interim between the original failure and the latest failures of this test, the logging was reduced from trace to info, so there is less info available in the latest failures than in the original ones. In the instances I've looked at, it has always been the n=1023,n_topics=1 variant of the test that's failing, which would support the possibility that this is resulting from an overloaded test system.

I'm going to set up a looping run of this test on clustered ducktape with trace logs enabled, to get a better sense of whether this is a real issue or a ghost from an overloaded docker environment.

[Lazin]

New occurrence https://buildkite.com/redpanda/redpanda/builds/9661#1702dd89-3269-4bff-8e2a-147f5bda5513/1471-9301

[dotnwat]

New occurence

• https://buildkite.com/redpanda/redpanda/builds/9717#d40868bb-65b8-4909-a395-bc165786327d
• k/fetch: validate fetch offset against high watermark #4201

Exact same message as in #4522 but it looks like that was closed as a duplicate of this one.

[dotnwat]

• https://buildkite.com/redpanda/redpanda/builds/9720#5b64fdb1-efb3-48fc-a1ce-c37b353ac146
• Added consumer group migration test when group topic is not present #4547

[twmb]

In #4583,
https://buildkite.com/redpanda/redpanda/builds/9788#a620f2eb-5f5c-4c82-a516-d500ccd23a13

[dotnwat]

moving this out the milestone. appears to be your friendly, routine ci-failure (that says hi a lot)

[jcsp]

We just need to get review on 👉 #4529

[abhijat]

seen on #4404

https://buildkite.com/redpanda/redpanda/builds/9821#8c6f4461-d5c8-41b0-a537-aecba2095564/1562-7648
https://buildkite.com/redpanda/redpanda/builds/9821#8724112e-2ecf-4a7d-a535-30f169973957/1561-7746