CI Failure "Segments not uploaded" in ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention

[VladLazar]

https://buildkite.com/redpanda/redpanda/builds/25512#018702b8-ca74-44fb-b6c7-1ef89bd17bb4

Module: rptest.tests.retention_policy_test
Class:  ShadowIndexingCloudRetentionTest
Method: test_cloud_time_based_retention
Arguments:
{
  "cloud_storage_type": 2
}

test_id:    rptest.tests.retention_policy_test.ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.ABS
status:     FAIL
run time:   24.909 seconds


    TimeoutError('Segments not uploaded')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/retention_policy_test.py", line 471, in test_cloud_time_based_retention
    wait_until(lambda: cloud_log_segment_count() == local_seg_count - 1,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Segments not uploaded

[VladLazar]

I think this test is just a bit flaky because it uses segment counts. We should update it to compare the local disk HWM with the uploaded start offset instead.

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/26632#01875a49-5c94-4995-9529-42e05de062f2

  8 FAIL test: ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.ABS (1/17 runs)
  9   failure at 2023-04-07T07:26:12.087Z: TimeoutError('Segments not uploaded')
 10       on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/26632#01875a49-5c94-4995-9529-42e05de062f2

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/27347#01879308-dcfd-465a-a7d8-1dcfac7eaf67

[NyaliaLui]

FAIL test: ShadowIndexingCloudRetentionTest.test_cloud_time_based_retention.cloud_storage_type=CloudStorageType.ABS (1/29 runs)
  failure at 2023-05-26T07:28:08.500Z: TimeoutError('Segments not uploaded')
      on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29955#018856bd-c63c-4eb4-b407-e5af69b790d7