Enabling maintenance mode on a decommissioned node leaves the cluster in an inconsistent state

[nicolaferraro]

What went wrong?

I'm trying maintenance mode hooks in combination with decommission and finding a strange behavior.

The problem is that maintenance mode can be enabled also for (fully) decommissioned nodes (static hooks do that automatically in k8s).
This causes two issues:

• The wait for maintenance mode to finish does not terminate as the finished=true state is never reached, as draining is not actually started (draining=false)
• Other nodes are unable to enter maintenance mode since the cluster believes that there's another node in maintenance mode, but it's decommissioned

I think it's ok if the cluster replies (200) ok to a request to enter maintenance mode for a decommissioned node, but probably the internal state remains inconsistent.

cc @dotnwat

Version & Environment

Redpanda v22.1.3, latest operator

How to reproduce the issue?

This process has been executed on a 3 node example cluster:

$ # Starting with a 3 nodes cluster (0, 1, 2), let's decommission broker 2
$ kubectl exec -c redpanda example-2 -- curl --silent -i -X PUT http://localhost:9644/v1/brokers/2/decommission
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: application/json
Date: Wed, 01 Jun 2022 17:18:12 GMT
Server: Seastar httpd

$ # Wait for broker 2 to disappear
$ kubectl exec -c redpanda example-0 -- curl --silent http://localhost:9644/v1/brokers/ | jq .[].node_id | sort
0
1

$ # Node 2 enters maintenance mode (this happens in practice because of static hooks currently set on pods)
$ kubectl exec -c redpanda example-2 -- curl --silent -i -X PUT http://localhost:9644/v1/brokers/2/maintenance
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: application/json
Date: Wed, 01 Jun 2022 17:21:22 GMT
Server: Seastar httpd

$ # Checking maintenance mode says it's not draining, while usually it says draining=true and finished=true
$ kubectl exec -c redpanda example-2 -- curl --silent http://localhost:9644/v1/maintenance
{"draining": false}

$ # Entering maintenance mode on another node fails
$ kubectl exec -c redpanda example-1 -- curl --silent -i -X PUT http://localhost:9644/v1/brokers/1/maintenance
HTTP/1.1 400 Bad Request
Content-Length: 93
Server: Seastar httpd
Date: Wed, 01 Jun 2022 17:23:51 GMT
Content-Type: application/json

{"message": "can not update broker 1 state, invalid state transition requested", "code": 400}

JIRA Link: CORE-933