admin: reject maintenance mode req on 1 node cluster #4921
Conversation
If a single node cluster puts its only node in maintenance mode, then there is no node elegible to become controller leader, and all further progress is stopped. Fixes redpanda-data#4338
If a system got into the bad state of issue redpanda-data#4338, then their cluster is broken until we replay the controller log _without_ putting the node into maintenance mode. Related redpanda-data#4338
jcsp
requested review from
dotnwat,
mmaslankaprv,
ztlpn and
VadimPlh
as code owners
| @@ -159,6 +159,17 @@ members_table::apply(model::offset version, maintenance_mode_cmd cmd) { | |||
| return errc::success; | |||
| } | |||
|
|
|||
| if (_brokers.size() < 2) { | |||
There was a problem hiding this comment.
Unrelated to this change per se, but do we also need a similar check when applying a decommission command? Maybe an even heavier-handed one, eg don't decommission to below the default replication factor, or don't decommission if there are single-replica partitions hosted on the affected node?
There was a problem hiding this comment.
Decommission will only finish if there are enough nodes in the cluster to keep requested topics replication factor. for single node cluster the decommission will only finish if another node will be added to the cluster.
There was a problem hiding this comment.
@andrwng yes, see the comment above "there is potentially an issue with decoms...". Basically shrinking below 3 is already broken, and needs handling as a separate job.
There was a problem hiding this comment.
Nice catch.
For more context: in a one node cluster if the node is put into maintenance mode then nothing bad should happen. leadership transfers won't occur, because they can't (there is no where to transfer the leadership). However if you are trying to reproduce this issue the problem will occur after restart. Maintenance mode prevents becoming leader after starting up too, so things like the controller or other raft groups won't elect themselves.
|
/backport v22.1.x |
|
Failed to run cherry-pick command. I executed the below command: |
Cover letter
Fixes #4338
Release notes
Improvements