assertion error in ShadowIndexingCacheSpaceLeakTest.test_si_cache

[andrwng]

CI failure: https://buildkite.com/redpanda/redpanda/builds/13576#01826743-26f1-40a3-9115-e9dd3810678b

====================================================================================================
test_id:    rptest.tests.test_si_cache_space_leak.ShadowIndexingCacheSpaceLeakTest.test_si_cache.message_size=10000.num_messages=100000.num_read=1000.concurrency=2
status:     FAIL
run time:   1 minute 9.344 seconds


    AssertionError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/test_si_cache_space_leak.py", line 115, in test_si_cache
    assert self._consumer.consumer_status.total_reads == num_read * concurrency
AssertionError

[abhijat]

#5934

https://buildkite.com/redpanda/redpanda/builds/13929#0182885d-f1e9-4264-adb7-a29b802dba0c

test_id: rptest.tests.test_si_cache_space_leak.ShadowIndexingCacheSpaceLeakTest.test_si_cache.message_size=10000.num_messages=100000.num_read=1000.concurrency=2

  | status: FAIL
  | run time: 14 minutes 4.752 seconds
  |  
  |  
  | TimeoutError("Timed out waiting 600 seconds for service nodes to finish. These nodes are still alive: ['FranzGoVerifiableRandomConsumer-0-140656854888224 node 1 on docker-rp-7']")
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
  | return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/test_si_cache_space_leak.py", line 112, in test_si_cache
  | self._consumer.wait()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 72, in wait
  | super(BackgroundThreadService, self).wait(timeout_sec)
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/services/service.py", line 267, in wait
  | raise TimeoutError("Timed out waiting %s seconds for service nodes to finish. " % str(timeout_sec)
  | ducktape.errors.TimeoutError: Timed out waiting 600 seconds for service nodes to finish. These nodes are still alive: ['FranzGoVerifiableRandomConsumer-0-140656854888224 node 1 on docker-rp-7']

[jcsp]

That original assertion assert self._consumer.consumer_status.total_reads == num_read * concurrency is not right. the consumer being used will loop until told to stop: if the intervening code takes long enough, the consumer will go round the loop and read 2x the number of messages that's being asserted here. It should be a >= assertion.