Crash: assertion in offset_translator_state (EndToEndTopicRecovery test_restore failed)

[LenaAn]

https://buildkite.com/redpanda/redpanda/builds/11728#0181a4d3-f70e-46e8-8299-0d796bdd5947

test_id:    rptest.tests.e2e_topic_recovery_test.EndToEndTopicRecovery.test_restore.message_size=10000.num_messages=100000.recovery_overrides=.retention.bytes.1024

status:     FAIL

run time:   6 minutes 14.865 seconds





    NodeCrash([(<ducktape.cluster.cluster.ClusterNode object at 0xffff88708610>, "ERROR 2022-06-27 11:57:55,423 [shard 1] assert - Assert failure: (../../../src/v/storage/offset_translator_state.cc:135) 'base_offset > rbegin->first' ntp {kafka/topic-pblgqsmnyx/0}: trying to add batch to offset translator at offset 9223372036854775807 that is not higher than the previous last offset 9223372036854775807\n"), (<ducktape.cluster.cluster.ClusterNode object at 0xffff88708c10>, "ERROR 2022-06-27 11:57:55,423 [shard 1] assert - Assert failure: (../../../src/v/storage/offset_translator_state.cc:135) 'base_offset > rbegin->first' ntp {kafka/topic-pblgqsmnyx/0}: trying to add batch to offset translator at offset 9223372036854775807 that is not higher than the previous last offset 9223372036854775807\n")])

Traceback (most recent call last):

  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped

    r = f(self, *args, **kwargs)

  File "/root/tests/rptest/tests/e2e_topic_recovery_test.py", line 156, in test_restore

    self._restore_topic(topic_spec, recovery_overrides)

  File "/root/tests/rptest/tests/e2e_topic_recovery_test.py", line 106, in _restore_topic

    rpk.describe_topic_configs(topic)

  File "/root/tests/rptest/clients/rpk.py", line 250, in describe_topic_configs

    output = self._run_topic(cmd)

  File "/root/tests/rptest/clients/rpk.py", line 421, in _run_topic

    return self._execute(cmd, stdin=stdin, timeout=timeout)

  File "/root/tests/rptest/clients/rpk.py", line 537, in _execute

    raise RpkException(

rptest.clients.rpk.RpkException: RpkException<command /var/lib/buildkite-agent/builds/arm64-xfs-builders-i-00e9cbf44a7a92cca-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/bin/rpk topic --brokers docker-rp-23:9092,docker-rp-19:9092,docker-rp-9:9092 describe topic-pblgqsmnyx -c returned 1, output:  error: unable to request configs: unable to dial: dial tcp 172.18.0.18:9092: connect: connection refused

>



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run

    data = self.run_test()

  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test

    return self.test_context.function(self.test)

  File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper

    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)

  File "/root/tests/rptest/services/cluster.py", line 38, in wrapped

    self.redpanda.raise_on_crash()

  File "/root/tests/rptest/services/redpanda.py", line 993, in raise_on_crash

    raise NodeCrash(crashes)

rptest.services.utils.NodeCrash: <NodeCrash (docker-rp-23,docker-rp-19) docker-rp-23: ERROR 2022-06-27 11:57:55,423 [shard 1] assert - Assert failure: (../../../src/v/storage/offset_translator_state.cc:135) 'base_offset > rbegin->first' ntp {kafka/topic-pblgqsmnyx/0}: trying to add batch to offset translator at offset 9223372036854775807 that is not higher than the previous last offset 9223372036854775807

[VladLazar]

Another failure here: https://buildkite.com/redpanda/redpanda/builds/12110#0181ce95-8115-4439-94f7-e03140cfc815.

[twmb]

https://buildkite.com/redpanda/redpanda/builds/11999#0181bcbe-ae71-4326-be80-624e21d41b24/2323-9216

[twmb]

Same PR, https://buildkite.com/redpanda/redpanda/builds/11999#0181bcbe-ae74-476a-a63a-bd9f846e9895/2497-9035

[LenaAn]

Another one https://buildkite.com/redpanda/redpanda/builds/12194#0181d4eb-befb-4923-abad-e93ab4209044 from #5364

[VadimPlh]

Same https://buildkite.com/redpanda/redpanda/builds/12198#0181d54d-a152-4309-abae-5831080423cb

[mmaslankaprv]

Seen again: https://buildkite.com/redpanda/redpanda/builds/12361#0181e6d7-2b0a-4dd1-980f-d51cd534d2a1

[NyaliaLui]

again https://buildkite.com/redpanda/redpanda/builds/12414#0181f105-067e-4845-82f2-cf48fcb274cc

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/12471#0181f501-72b2-46e2-8b9f-05ec643b7c9d

[NyaliaLui]

Same test but different failure. I'm marking it here because it seems related
https://buildkite.com/redpanda/redpanda/builds/12511#0181f85e-2379-494e-8a8b-28fc629a1d95/1634-8782

[VadimPlh]

Again https://buildkite.com/redpanda/redpanda/builds/12977#01822ed4-bd8e-45cf-9f83-aba57b5e903e

[VadimPlh]

Again https://buildkite.com/redpanda/redpanda/builds/13005#018234dc-3b92-4481-ad26-442f76c985b5

[Lazin]

This should be fixed now by #5544
The offset translator assertion is triggered by the empty segments (recovery removes config batches and they can become empty).

[BenPope]

Saw this on v21.1.x: https://buildkite.com/redpanda/redpanda/builds/16459#0183c6d5-beaa-4b01-bdbe-dd8ca1e8aeff.

@Lazin I guess #5544 is too big to backport?

[BenPope]

Seen on v22.1.x: https://buildkite.com/redpanda/vtools/builds/3885#01840924-eca0-4bfe-96ba-84ab237275cb

[jcsp]

I think we probably won't do a bespoke version of #5544 for 22.1.x unless this becomes an issue in the field