Failure in NodesDecommisioningTest.test_decommissioning_working_node

[jcsp]

https://buildkite.com/vectorized/vtools/builds/306#3d5fa10a-5652-48fe-8e90-28832108c106

http://ci-artifacts.dev.vectorized.cloud/vtools/3d5fa10a-5652-48fe-8e90-28832108c106/vbuild/ducktape/results/2021-09-22--001/report.html

From the test log, it looks like the partitions were correctly moved away from the node, but the node itself remained in the list of brokers and the test timed out waiting for it to go away.

[jcsp]

This looks like a genuine redpanda bug.

On a passing run of the test, the time between "changing node {} membership state to: draining" and "decommissioning finished, removing node {} from cluster" is only 10 seconds.

On this failure, we're never getting the second message, and the timeout is 120 seconds.

I do notice that the leadership balancer is doing transfers at the same time as the decommissioning is going on.

[jcsp]

This test is now disabled in dev, PR that fixes it should re-enable test too.

[jcsp]

This is hard to reproduce. Has not failed in several days of nightly test-staging runs. The original failure still looks like an authentic issue.

[jcsp]

Here's a fresh failure from test-staging:
https://buildkite.com/vectorized/redpanda/builds/2705#a32401c8-50c4-4f64-951a-86897f3c5ccf

[jcsp]

Created #2478 for what is believed to be the underlying bug.

[jcsp]

Should be fixed by #3125

[jcsp]

Fixed via #3125 via #2478