k8s: Put brokers in maintenance mode before deleting orphan's pod #7530
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
3 times, most recently
from
1bd32aa
to
11470aa
Compare
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
from
11470aa
to
11db9ee
Compare
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
2 times, most recently
from
8fc043f
to
d6df2bd
Compare
alenkacz
reviewed
Jan 2, 2023
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
3 times, most recently
from
3ca5de5
to
86045b8
Compare
alenkacz
previously approved these changes
Jan 3, 2023
| return &RequeueAfterError{RequeueAfter: RequeueDuration, Msg: "wait for pod restart"} | ||
| } | ||
|
|
||
| //nolint:goerr113 // out of scope for this PR |
There was a problem hiding this comment.
unrelated to this PR: let's finally disable this linter 🙏 :D
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
from
86045b8
to
13dd38f
Compare
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
from
13dd38f
to
8d89529
Compare
nicolaferraro
previously approved these changes
Jan 4, 2023
alenkacz
previously approved these changes
Jan 4, 2023
RafalKorepta
dismissed stale reviews from alenkacz and nicolaferraro
via
eae69eb
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
from
8d89529
to
eae69eb
Compare
During rolling update, before this change, Redpanda operator was calculating the difference between running pod specification and stateful set pod template. If the specification did not match the pod was deleted. From release v22.1.1 operator is configuring each broker with pod lifecycle hooks. In the PreStop hook the script will try to put broker into maintenance mode for 120 seconds before POD is terminated. Redpanda could not finish within 120 seconds to put one broker into maintenance mode. This PR improves the situation by putting maintenance mode before POD is deleted. The `EnableMaintanaceMode` function is called multiple times until `Broker` function returns correct status. The assumption is that REST admin API maintenance mode endpoint is idempotent. When pod is successfully deleted statefulset would reschedule the pod with correct pod specification. redpanda-data#4125 redpanda-data#3023
RafalKorepta
force-pushed
the
rk/gh-3023/put-in-maintanance-mode
branch
from
eae69eb
to
3c34855
Compare
nicolaferraro
approved these changes
Jan 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During rolling update, before this change, Redpanda operator was calculating the difference between running pod specification and stateful set pod template. If the specification did not match the pod was deleted. From release v22.1.1 operator is configuring each broker with pod lifecycle hooks. In the PreStop hook the script will try to put broker into maintenance mode for 120 seconds before POD is terminated. Redpanda could not finish within 120 seconds to put one broker into maintenance mode.
This PR improves the situation by putting maintenance mode before POD is deleted. The
putInMaintenanceModefunction is called multiple times untilBrokerfunction returns correct status. The assumption is that REST admin API maintenance mode endpoint is idempotent.When pod is successfully deleted statefulset would reschedule the pod with correct pod specification.
Ref
#4125
#3023
On top of:
#7528
Backports Required
UX Changes
Before deleting the POD maintenance mode is configured (not within 120 second lifecycle hook).
Release Notes
Improvements