topic_recovery_test.TopicRecoveryTest.test_missing_segment

[Lazin]

https://buildkite.com/redpanda/redpanda/builds/9621#08fea5ac-6df1-46f5-b16d-8d7834a987e5/1554-7717

test_id:    rptest.tests.topic_recovery_test.TopicRecoveryTest.test_missing_segment
--
  | status:     FAIL
  | run time:   1 minute 10.230 seconds
  |  
  |  
  | AssertionError("NTP NTP(ns='kafka', topic='panda-topic', partition=0) the restored partition is larger 4084150 than the original one 4063335.")
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 1223, in test_missing_segment
  | self.do_run(test_case)
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 1174, in do_run
  | test_case.validate_cluster(baseline, restored)
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 614, in validate_cluster
  | _verify_file_layout(
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 159, in _verify_file_layout
  | assert rest_ntp_size <= orig_ntp_size, f"NTP {ntp} the restored partition is larger {rest_ntp_size} than the original one {orig_ntp_size}."
  | AssertionError: NTP NTP(ns='kafka', topic='panda-topic', partition=0) the restored partition is larger 4084150 than the original one 4063335.

[mmaslankaprv]

Seen again: https://buildkite.com/redpanda/redpanda/builds/9664#1a4fc04c-7fb8-4b65-bc37-65082c096675

[mmaslankaprv]

@Lazin should we be worried about this ?

[dotnwat]

seen again here https://github.com/redpanda-data/vtools/pull/631

[dotnwat]

[Michal [Yesterday at 3:30 AM]
I have a quick question about: https://github.com//issues/4520. Should this block v22.1.1 release ?

[Evgeny [1 day ago]
I don’t think so. It looks like this isn’t a bug in redpanda but a test problem (test doesn’t account for preallocation of disk space).

not a blocker for release. removing from milestone.

[abhijat]

seen on #4404
https://buildkite.com/redpanda/redpanda/builds/9821#58ccc865-daf9-4429-bbe0-563e8835c70a/1527-7630

[rystsov]

Another instance: https://buildkite.com/redpanda/redpanda/builds/9879#3942caa5-dde9-4623-a995-b5b5f587d6e5

test_id:    rptest.tests.topic_recovery_test.TopicRecoveryTest.test_missing_segment
status:     FAIL
run time:   54.016 seconds


    AssertionError("NTP NTP(ns='kafka', topic='panda-topic', partition=0) the restored partition is larger 4226841 than the original one 4199948.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 1223, in test_missing_segment
    self.do_run(test_case)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 1174, in do_run
    test_case.validate_cluster(baseline, restored)
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 614, in validate_cluster
    _verify_file_layout(
  File "/root/tests/rptest/tests/topic_recovery_test.py", line 159, in _verify_file_layout
    assert rest_ntp_size <= orig_ntp_size, f"NTP {ntp} the restored partition is larger {rest_ntp_size} than the original one {orig_ntp_size}."
AssertionError: NTP NTP(ns='kafka', topic='panda-topic', partition=0) the restored partition is larger 4226841 than the original one 4199948.

[abhijat]

fix wip here

[BenPope]

This may have returned: https://buildkite.com/redpanda/vtools/builds/3183#01828adc-074b-45e5-b224-1245d61997dc

Module: rptest.tests.topic_recovery_test
Class:  TopicRecoveryTest
Method: test_fast3

[INFO  - 2022-08-11 04:57:53,999 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.topic_recovery_test.TopicRecoveryTest.test_fast3: FAIL: AssertionError("NTP NTP(ns='kafka', topic='panda-topic-2', partition=0) the restored partition is larger 5265327 than the original one 5264821.")
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/topic_recovery_test.py", line 1300, in test_fast3
    self.do_run(test_case)
  File "/home/ubuntu/redpanda/tests/rptest/tests/topic_recovery_test.py", line 1186, in do_run
    test_case.validate_cluster(baseline, restored)
  File "/home/ubuntu/redpanda/tests/rptest/tests/topic_recovery_test.py", line 667, in validate_cluster
    verify_file_layout(baseline,
  File "/home/ubuntu/redpanda/tests/rptest/utils/si_utils.py", line 199, in verify_file_layout
    assert rest_ntp_size <= orig_ntp_size, \
AssertionError: NTP NTP(ns='kafka', topic='panda-topic-2', partition=0) the restored partition is larger 5265327 than the original one 5264821.

[abhijat]

it looks like we are now erring on the other side, IE restored is larger than original, I wonder if the recent changes to restore segment and the way we deduct config batches from original partition are now conflicting with each other.