Init Cluster UUID in upgraded clusters #7079
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrwng
reviewed
Nov 3, 2022
andrwng
reviewed
Nov 3, 2022
andrwng
reviewed
Nov 4, 2022
dlex
force-pushed
the
333_cluster-uuid-after-upgrade
branch
from
1f01d0c
to
7cdaad1
Compare
|
Force-push: rebased to a fresh |
andrwng
reviewed
Nov 4, 2022
dlex
force-pushed
the
333_cluster-uuid-after-upgrade
branch
from
7cdaad1
to
f93e0c3
Compare
|
force push: |
andrwng
reviewed
Nov 5, 2022
A `register_route` overload to register synchronous handlers Storage to get cluster_uuid is accessed via controller
Check that cluster_uuid is not available until the entire cluster upgraded, and after the upgrade is complete that cluster_uuid is here
New feature `seeds_driven_bootstrap_capable` to control cluster upgrade. Non-founding raft0 leader issues `bootstrap_cluster_cmd` if no cluster_uuid present. This is done in a detached fiber waiting on the new feature to be enabled. Controller log existence checks based on raft visible index have been removed because they are prone to a race condition.
A node that is already a member of a cluster failed to apply additional properties of bootstrap_cluster_cmd like bootstrap user upon restarts because idempotency criterion was to check presence of cluster_uuid value restored from kvstore. Now bootstrap_backend has its own copy that reflects only the fact of the command application. Also reconciliation between the two has been added.
Successful bootstrap user creation caused bootstrap_cluster_cmd to fail early and never apply cluster_uuid. Now user_exists condition is a warning and cluster_uuid creation is completed in that case together with success. All other (fallback) user creation errors are fatal.
to agree on using "cluster UUID" as the common label, and on the levels to produce reasonable output volume at INFO.
to provide a label for them distinct from cluster UUID's one
so that it's empty if and only if we are not a cluster founder
dlex
force-pushed
the
333_cluster-uuid-after-upgrade
branch
from
f93e0c3
to
903ad6f
Compare
|
force-push: fixed issues caused by race condition between controller and the raft0 consensus leading to nondeterministic results on whether controller log exists or not, and issues around bootstrap user creation; also some log factoring |
|
Known CI failures: |
andrwng
approved these changes
Nov 8, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
When an existing cluster is upgraded to 22.3, we want it to have the
bootstrap_cluster_cmdin the controller log, as well as a cluster UUID assigned. This way going forward we will be able to rely on cluster UUID as a mandatory property of any existing cluster.This PR also adds an admin API endpoint
v1/cluster/uuidso that it is possible to query the cluster UUID a node belongs to.Fixes #333
Backport Required
UX changes
Admin API
v1/cluster/uuidreturns{"cluster_uuid": "<UUID>"}or empty if the node is not a part of a cluster yet.Release notes