test: rptest.tests.topic_delete_test.TopicDeleteStressTest.stress_test unclean client shutdown

[dotnwat]

when looking into the logs around the time that the badlogline was reported, it seems as though ducktape may have kill -9 on a kafka client which could cause this issue. it may be that we need to improve unclean shutdown.

this looks like a kafka client terminated or crashed in the test.

test_id:    rptest.tests.topic_delete_test.TopicDeleteStressTest.stress_test
--
  | status:     FAIL
  | run time:   6 minutes 15.546 seconds
  |  
  |  
  | <BadLogLines nodes=docker_n_13(1) example="ERROR 2022-04-19 18:24:06,405 [shard 0] rpc - server.cc:114 - kafka rpc protocol - Error[applying protocol] remote address: 172.18.0.10:55738 - std::out_of_range (Invalid skip(n). Expected:999597, but skipped:996533)">
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/services/cluster.py", line 47, in wrapped
  | self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  | File "/root/tests/rptest/services/redpanda.py", line 849, in raise_on_bad_logs
  | raise BadLogLines(bad_lines)
  | rptest.services.redpanda.BadLogLines: <BadLogLines nodes=docker_n_13(1) example="ERROR 2022-04-19 18:24:06,405 [shard 0] rpc - server.cc:114 - kafka rpc protocol - Error[applying protocol] remote address: 172.18.0.10:55738 - std::out_of_range (Invalid skip(n). Expected:999597, but skipped:996533)">
  |  

Originally posted by @dotnwat in #4199 (comment)

[dotnwat]

@rystsov when I disable idempotence this problem appears to go away.

diff --git a/src/v/config/configuration.cc b/src/v/config/configuration.cc
index e98586cb5..98dd96e94 100644
--- a/src/v/config/configuration.cc
+++ b/src/v/config/configuration.cc
@@ -415,7 +415,7 @@ configuration::configuration()
       "enable_idempotence",
       "Enable idempotent producer",
       {.visibility = visibility::user},
-      true)
+      false)
   , enable_transactions(
       *this,
       "enable_transactions",

The message std::out_of_range (Invalid skip(n). Expected:999597, but skipped:996533) is often associated with a client that crashes or stops uncleanly. Though, it could potentially be due to other things. I haven't investigated further than reproducing w/ and w/o idempotence.

[jcsp]

This failed 7/54 runs in the last 24h, last example was https://buildkite.com/redpanda/redpanda/builds/9256#138b41aa-c3a9-439a-a21f-c113c5af1636

I was about to mark it ok_to_fail, but I think it's better to just make it temporarily tolerant of this error message
#4367

[dotnwat]

@rystsov: not a release blocker moving to next patch release

[rystsov]

#5238 introduces a new workload & online verifier which isn't subject to such errors when it's it we switch to it and it will fix this issue

[dotnwat]

#5238 introduces a new workload & online verifier which isn't subject to such errors when it's it we switch to it and it will fix this issue

nice!

[dotnwat]

@rystsov can you decide if this important, fixed, etc...?

[jcsp]

No repros in 30d