out_of_range failure in FranzGoVerifiableWithSiTest.test_si_with_timeboxed.segment_size=20971520

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/9879#8e5893f2-3651-4574-b6fb-023068c913a4

[WARNING - 2022-05-07 07:00:12,068 - redpanda - raise_on_bad_logs - lineno:1056]: [test_si_with_timeboxed] Unexpected log line on docker-rp-12: ERROR 2022-05-07 06:56:56,436 [shard 1] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.18.0.40:37162 - std::out_of_range (Invalid skip(n). Expected:1000097, but skipped:800508)

[INFO:2022-05-07 07:00:12,482]: RunnerClient: rptest.tests.franz_go_verifiable_test.FranzGoVerifiableWithSiTest.test_si_with_timeboxed.segment_size=20971520: FAIL: <BadLogLines nodes=docker-rp-12(1) example="ERROR 2022-05-07 06:56:56,436 [shard 1] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.18.0.40:37162 - std::out_of_range (Invalid skip(n). Expected:1000097, but skipped:800508)">

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 47, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1071, in raise_on_bad_logs
    raise BadLogLines(bad_lines)

rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-12(1) example="ERROR 2022-05-07 06:56:56,436 [shard 1] rpc - server.cc:116 - kafka rpc protocol - Error[applying protocol] remote address: 172.18.0.40:37162 - std::out_of_range (Invalid skip(n). Expected:1000097, but skipped:800508)">

[twmb]

https://buildkite.com/redpanda/vtools/builds/2172#edca8b10-d923-4c56-b971-a523efb58926

[mmaslankaprv]

Another failure mode of this test:

WARN  2022-05-10 21:38:28,126 [shard 1] cloud_storage - [fiber52~0 kafka/topic-qnvgcdmcft/93] - remote_segment.cc:262 - Failed to build a remote_segment index, error: std::bad_alloc (std::bad_alloc)
redpanda: /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:150: void seastar::future_state_base::ignore(): Assertion `0 && "invalid state for ignore"' failed.
``

[jcsp]

@mmaslankaprv there are a couple of tickets related to this test, the bad_alloc is #4626