Assertion Error in TopicRecoveryTest.test_fast3

[r-vasquez]

Saw in: https://buildkite.com/redpanda/redpanda/builds/13268#_

test_id:    rptest.tests.topic_recovery_test.TopicRecoveryTest.test_fast3
--
  | status:     FAIL
  | run time:   2 minutes 31.334 seconds
  |  
  |  
  | AssertionError()
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 1219, in test_fast3
  | self.do_run(test_case)
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 1121, in do_run
  | test_case.after_restart_validation()
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 613, in after_restart_validation
  | self._produce_and_verify(topic)
  | File "/root/tests/rptest/tests/topic_recovery_test.py", line 188, in _produce_and_verify
  | assert new_hw is not None
  | AssertionError

[abhijat]

the high watermark is supposed to come from rpk topic describe which returned NOT_LEADER_FOR_PARTITION

[DEBUG - 2022-07-29 18:08:38,819 - rpk - _execute - lineno:539]: Executing command: ['/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-05aa9624bd7904f95-1/redpanda/redpanda/vbuild/debug/clang/dist/local/redpanda/bin/rpk', 'topic', '--brokers', 'docker-rp-18:9092,docker-rp-17:9092,docker-rp-5:9092', 'describe', 'panda-topic-1', '-p']
[DEBUG - 2022-07-29 18:08:38,871 - rpk - _execute - lineno:552]: PARTITION  LEADER  EPOCH  REPLICAS  LOG-START-OFFSET          HIGH-WATERMARK
0          1       4      [1 2 3]   NOT_LEADER_FOR_PARTITION  NOT_LEADER_FOR_PARTITION
1          3       5      [1 2 3]   NOT_LEADER_FOR_PARTITION  NOT_LEADER_FOR_PARTITION

[r-vasquez]

Just happened in https://buildkite.com/redpanda/redpanda/builds/14085#018293a4-8447-4943-b185-7d601b019a43 It's in v22.2.x branch

#6004

[dotnwat]

@r-vasquez looks like this was closed yesterday and your report came in 21 hours ago. do you think that maybe the fix wasn't merged when your test ran?

[dotnwat]

cc @abhijat

[abhijat]

The failed assertion seen in https://buildkite.com/redpanda/redpanda/builds/14085#018293a4-8447-4943-b185-7d601b019a43:

assert new_hw is not None

is from the old version of the test. This assertion is no longer there in code with this PR. I have created a backport PR for v22.2.x and v22.1.x branches.