-
Notifications
You must be signed in to change notification settings - Fork 577
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
k8s: Put brokers in maintenance mode before deleting orphant pod
During rolling update, before this change, Redpanda operator was calculating the difference between running pod specification and stateful set pod template. If the specification did not match the pod was deleted. From release v22.1.1 operator is configuring each broker with pod lifecycle hooks. In the PreStop hook the script will try to put broker into maintenance mode for 120 seconds before POD is terminated. Redpanda could not finish within 120 seconds to put one broker into maintenance mode. This PR improves the situation by putting maintenance mode before POD is deleted. The `EnableMaintanaceMode` function is called multiple times until `Broker` function returns correct status. The assumption is that REST admin API maintenance mode endpoint is idempotent. When pod is successfully deleted statefulset would reschedule the pod with correct pod specification. #4125 #3023
- Loading branch information
Rafal Korepta
committed
Jan 5, 2023
1 parent
29d348a
commit 3c34855
Showing
7 changed files
with
260 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
// Copyright 2022-2023 Redpanda Data, Inc. | ||
// | ||
// Use of this software is governed by the Business Source License | ||
// included in the file licenses/BSL.md | ||
// | ||
// As of the Change Date specified in that file, in accordance with | ||
// the Business Source License, use of this software will be governed | ||
// by the Apache License, Version 2.0 | ||
|
||
package utils_test | ||
|
||
import ( | ||
"fmt" | ||
"testing" | ||
|
||
"github.com/redpanda-data/redpanda/src/go/k8s/pkg/utils" | ||
"github.com/stretchr/testify/require" | ||
) | ||
|
||
func TestGetPodOrdinal(t *testing.T) { | ||
tcs := []struct { | ||
podName string | ||
clusterName string | ||
expectedError bool | ||
expectedOrdinal int64 | ||
}{ | ||
{"", "", true, -1}, | ||
{"test", "", true, -1}, | ||
{"pod-0", "pod", false, 0}, | ||
{"pod-99", "pod", false, 99}, | ||
{"", "unexpected longer cluster name", true, -1}, | ||
{"test+0", "test", false, 0}, | ||
{"without-ordinal-", "without-ordinal", true, -1}, | ||
} | ||
|
||
for _, tc := range tcs { | ||
t.Run(fmt.Sprintf("pod %s and cluster %s", tc.podName, tc.clusterName), func(t *testing.T) { | ||
ordinal, err := utils.GetPodOrdinal(tc.podName, tc.clusterName) | ||
if tc.expectedError { | ||
require.Error(t, err) | ||
} | ||
require.Equal(t, tc.expectedOrdinal, ordinal) | ||
}) | ||
} | ||
} |