CI Failure (heap-use-after-free) in TxFeatureFlagTest.test_disabling_transactions_after_they_being_used

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/15094#018317d5-4e4a-4a52-8bad-9de3f437e31b

Module: rptest.tests.tx_feature_flag_test
Class:  TxFeatureFlagTest
Method: test_disabling_transactions_after_they_being_used

test_id:    rptest.tests.tx_feature_flag_test.TxFeatureFlagTest.test_disabling_transactions_after_they_being_used
status:     FAIL
run time:   2 minutes 40.395 seconds

    <BadLogLines nodes=docker-rp-5(1) example="==5904==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000061178 at pc 0x555efabb9a1e bp 0x7fc74fcd4df0 sp 0x7fc74fcd4de8">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 51, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1326, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-5(1) example="==5904==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000061178 at pc 0x555efabb9a1e bp 0x7fc74fcd4df0 sp 0x7fc74fcd4de8">

WARN  2022-09-07 13:23:52,274 [shard 1] tx - rm_partition_frontend.cc:541 - got error rpc::errc:1 on remote commit tx
TRACE 2022-09-07 13:23:52,275 [shard 1] tx - rm_partition_frontend.cc:516 - received name:commit_tx, ntp:{kafka/tx-topic/0}, pid:{producer_identity: id=1, epoch=0}, tx_seq:1, ec:cluster::tx_errc:7
TRACE 2022-09-07 13:23:52,275 [shard 1] tx - tx_gateway_frontend.cc:49 - released_lock name:end_txn, tx_id:{test-tx-id}
=================================================================
==5904==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000061178 at pc 0x555efabb9a1e bp 0x7fc74fcd4df0 sp 0x7fc74fcd4de8
READ of size 8 at 0x60b000061178 thread T1 (reactor-1)
    #0 0x555efabb9a1d  (/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-01fd84467127f7de6-1/redpanda/redpanda/vbuild/debug/clang/dist/local/redpanda/libexec/redpanda+0x346c9a1d) (BuildId: debf613c9119170a95d560adca3f3915877dae2c)
    #1 0x555efae84607  (/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-01fd84467127f7de6-1/redpanda/redpanda/vbuild/debug/clang/dist/local/redpanda/libexec/redpanda+0x34994607) (BuildId: debf613c9119170a95d560adca3f3915877dae2c)
...

[dlex]

One more: https://buildkite.com/redpanda/redpanda/builds/15084#0183150a-1f35-4bc9-8029-4a372902deb9

[dlex]

Stacktraces:
redpanda.log.access.stack.txt
redpanda.log.alloc.stack.txt
redpanda.log.free.stack.txt

[rystsov]

The fix didn't helped: https://buildkite.com/redpanda/redpanda/builds/15232#01833348-60a9-4595-9c24-826b7dbb5b24

[VladLazar]

New instance on 20.09.2022:
https://buildkite.com/redpanda/redpanda/builds/15422#018349db-92b7-49eb-9dfd-86165d7b1cf3

[VladLazar]

Another one on 21.09.2022:
https://buildkite.com/redpanda/redpanda/builds/15551#01835e89-3270-4bb7-82f0-ed87db213717

[jcsp]

Same
https://buildkite.com/redpanda/redpanda/builds/15710#01837966-9d28-4a71-9d6a-5d65fc9d7aac