-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot disable maintenance mode after restart #4338
Comments
@dotnwat is it possible that maintenance mode is causing the node to refuse all leaderships including controller leadership? We probably need a special case for single node clusters if that's the case |
@jcsp yes, maintenance mode tries to prevent all leadership, including the controller. we can certainly special case the controller, but i didn't do this up front because I wasn't imagining any scenario in which it would be problematic, but moving it would be beneficial for any clients in the midst of doing administrative stuff (or internal communication with the controller leader).
hmm, ill take a look. we may need some retries, unless i'm missing something else here in the ticket. |
If a single node cluster puts its only node in maintenance mode, then there is no node elegible to become controller leader, and all further progress is stopped. Fixes redpanda-data#4338
If a system got into the bad state of issue redpanda-data#4338, then their cluster is broken until we replay the controller log _without_ putting the node into maintenance mode. Related redpanda-data#4338
If a single node cluster puts its only node in maintenance mode, then there is no node elegible to become controller leader, and all further progress is stopped. Fixes redpanda-data#4338 (cherry picked from commit b970b2d)
If a system got into the bad state of issue redpanda-data#4338, then their cluster is broken until we replay the controller log _without_ putting the node into maintenance mode. Related redpanda-data#4338 (cherry picked from commit 20f3597)
Issue description
There's an issue with disabling maintenance mode after a pod is restarted with maintenance mode on. I think this has been introduced recently as e2e tests were working in #4125.
This is the error I get from rpk:
I extracted a reproducer that can be run on Kubernetes. It's happening on a single node cluster, but may happen also with multiple instances.
Base resource:
How to reproduce the issue?
kubectl apply -f example.yaml
kubectl exec example-0 -- rpk cluster maintenance enable 0
kubectl delete pod example-0
kubectl exec example-0 -- rpk cluster maintenance disable 0
Then the error is returned.
I believe it`s due to some recent changes on management of leadership groups, but I'm not sure. cc: @dotnwat .
The cluster node is reporting errors as well:
The text was updated successfully, but these errors were encountered: