CI Failure (restored partition is too small) in TopicRecoveryTest.test_fast2

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/15179#01832351-c1ef-4ee2-920f-ac41ebbb79d7

Module: rptest.tests.topic_recovery_test
Class:  TopicRecoveryTest
Method: test_fast2

test_id:    rptest.tests.topic_recovery_test.TopicRecoveryTest.test_fast2
status:     FAIL
run time:   1 minute 23.901 seconds

    AssertionError("NTP NTP(ns='kafka', topic='panda-topic-2', partition=2) the restored partition is too small 3533238. The original is 3550365 bytes which 17127 bytes larger.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 1306, in test_fast2
    self.do_run(test_case)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 1212, in do_run
    test_case.validate_cluster(baseline, restored)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 693, in validate_cluster
    verify_file_layout(baseline,
  File "/root/tests/rptest/utils/si_utils.py", line 204, in verify_file_layout
    assert delta <= BLOCK_SIZE, \
AssertionError: NTP NTP(ns='kafka', topic='panda-topic-2', partition=2) the restored partition is too small 3533238. The original is 3550365 bytes which 17127 bytes larger.

[rystsov]

The issue was previously discussed in #4960. Also it seems that it has the same root cause as #4887

[mmedenjak]

Also it seems that it has the same root cause as #4887

From Evgeny:

looks like two different issues for me (timeout issue could probably be solved by simply downsizing the data set since it generates a lot of segments)

[jcsp]

Last failure of this kind was:
failure at 2022-10-02T09:35:22.985Z: AssertionError("NTP NTP(ns='kafka', topic='panda-topic-2', partition=1) the restored partition is too small 3469188. The original is 3496805 bytes which 27617 bytes larger.")
in job https://buildkite.com/redpanda/vtools/builds/3627#018396a6-afdf-4dae-8a44-ddac93c4a005

[NyaliaLui]

Possible related failure in https://buildkite.com/redpanda/redpanda/builds/18184#01845ac8-1a7f-4763-b63e-e53ec491599c/6-678

    AssertionError('old_hw 3319 unexpectedly same as new_hw 3319 for topic spec: panda-topic-1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 1352, in test_fast2
    self.do_run(test_case)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 1258, in do_run
    test_case.validate_cluster(baseline, restored)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 706, in validate_cluster
    self._produce_and_verify(topic)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 219, in _produce_and_verify
    assert old_state.hwm != new_state.hwm, \
AssertionError: old_hw 3319 unexpectedly same as new_hw 3319 for topic spec: panda-topic-1

[jcsp]

The failure in this ticket's OP has not reoccurred in the last 30 days. Opened #7214 for the failure that was seen yesterday.