forked from redpanda-data/redpanda
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
k8s: Put brokers in maintenance mode before deleting orphant pod
During rolling update, before this change, Redpanda operator was calculating the difference between running pod specification and stateful set pod template. If the specification did not match the pod was deleted. From release v22.1.1 operator is configuring each broker with pod lifecycle hooks. In the PreStop hook the script will try to put broker into maintenance mode for 120 seconds before POD is terminated. Redpanda could not finish within 120 seconds to put one broker into maintenance mode. This PR improves the situation by putting maintenance mode before POD is deleted. The `EnableMaintanaceMode` function is called multiple times until `Broker` function returns correct status. The assumption is that REST admin API maintenance mode endpoint is idempotent. When pod is successfully deleted statefulset would reschedule the pod with correct pod specification. redpanda-data#4125 redpanda-data#3023
- Loading branch information
Rafal Korepta
committed
Jan 3, 2023
1 parent
15bde9d
commit 13dd38f
Showing
6 changed files
with
177 additions
and
31 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
// Copyright 2022-2023 Redpanda Data, Inc. | ||
// | ||
// Use of this software is governed by the Business Source License | ||
// included in the file licenses/BSL.md | ||
// | ||
// As of the Change Date specified in that file, in accordance with | ||
// the Business Source License, use of this software will be governed | ||
// by the Apache License, Version 2.0 | ||
|
||
package utils_test | ||
|
||
import ( | ||
"fmt" | ||
"testing" | ||
|
||
"github.com/redpanda-data/redpanda/src/go/k8s/pkg/utils" | ||
"github.com/stretchr/testify/require" | ||
) | ||
|
||
func TestGetPodOrdinal(t *testing.T) { | ||
tcs := []struct { | ||
podName string | ||
clusterName string | ||
expectedError bool | ||
expectedOrdinal int64 | ||
}{ | ||
{"", "", true, -1}, | ||
{"test", "", true, -1}, | ||
{"pod-0", "pod", false, 0}, | ||
{"pod-99", "pod", false, 99}, | ||
{"", "unexpected longer cluster name", true, -1}, | ||
{"test+0", "test", false, 0}, | ||
{"without-ordinal-", "without-ordinal", true, -1}, | ||
} | ||
|
||
for _, tc := range tcs { | ||
t.Run(fmt.Sprintf("pod %s and cluster %s", tc.podName, tc.clusterName), func(t *testing.T) { | ||
ordinal, err := utils.GetPodOrdinal(tc.podName, tc.clusterName) | ||
if tc.expectedError { | ||
require.Error(t, err) | ||
} | ||
require.Equal(t, tc.expectedOrdinal, ordinal) | ||
}) | ||
} | ||
} |