failure in rptest.tests.wasm_partition_movement_test.WasmPartitionMovementTest.test_dynamic_with_failure

[mmaslankaprv]

https://buildkite.com/redpanda/redpanda/builds/8271#d770bb1f-c833-468b-801a-6c5e98789416

One of the nodes has crashed:

TRACE 2022-03-21 15:00:41,540 [shard 2] kafka - fetch.cc:339 - fetch_ntps_in_parallel: for 1 partitions returning 0 total bytes
libc++abi: terminating
ERROR 2022-03-21 15:00:41,540 [shard 0] rpc - Error dispatching client reads: std::__1::system_error (error system:104, read: Connection reset by peer)
ERROR 2022-03-21 15:00:41,540 [shard 0] coproc - event_handler.cc:39 - Wasm engine failed to reply to heartbeat within the expected interval
Aborting on shard 2.
Backtrace:
  0x3809ce7
  linux-vdso.so.1+0x667
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc.so.6+0x3884f
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc.so.6+0x2475b
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc++abi.so.1+0x2a9e3
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc++abi.so.1+0xec43
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc++abi.so.1+0x29b6f
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc++abi.so.1+0x29b03
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc++.so.1+0x4649b
  0x1e38367
  0x37c5547
  0x37c7f53
  0x380bd7f
  0x377ec3f
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libpthread.so.0+0x7ff7
  /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0d2a70ef27c913b9c-1/redpanda/redpanda/vbuild/release/clang/dist/local/redpanda/lib/libc.so.6+0xdb35b

[graphcareful]

[INFO  - 2022-03-21 15:01:13,183 - runner_client - log - lineno:266]: RunnerClient: rptest.tests.wasm_partition_movement_test.WasmPartitionMovementTest.test_dynamic_with_failure: Summary: <BadLogLines nodes=docker_n_20(3) example="ERROR 2022-03-21 15:00:44,675 [shard 1] storage - Cannot continue parsing. recived size:272 bytes, expected:3964 bytes.

Looks like storage is failing for some reason, I see this error occur 3 times on this node, has anyone seen this before?

[mmaslankaprv]

storage reported error beacuse node crashed before, it is reporting data corruption at the end of a segment which is expected. Tests do not inject hard failures but stop redpanda gracefully, the problem is caused by the fact that redpanda crashes while shutting down.

[VadimPlh]

see again https://ci-artifacts.dev.vectorized.cloud/redpanda/6c964c53-afd4-4a49-b1e2-f340a9a17441/vbuild/ducktape/results/2022-04-04--001/report.txt

[dotnwat]

closing this ci-failure ticket. the test is disabled, and there is no scheduled coproc work/maintenance. in the meantime it is polluting the ci-failure clean-up effort statistics.