🐛 Ensure infrastructure is destroyed before deleting the node #2570
Conversation
k8s-ci-robot
added
cncf-cla: yes
enxebre
changed the title
|
/milestone v0.3.x |
k8s-ci-robot
added
size/L
|
/hold Holding for now to wait after |
k8s-ci-robot
added
the
do-not-merge/hold
If there were an edge case where we deleted an unreachable node and therefore removing any pod from etcd, but an stateful process were still running on the underlying host, we'll be immediately freeing up the pod name from the apiserver. "This would let the StatefulSet controller create a replacement Pod with that same identity; this can lead to the duplication of a still-running Pod, and if said Pod can still communicate with the other members of the StatefulSet, will violate the at most one semantics that StatefulSet is designed to guarantee." To mitigate this it'd be safer to delete the node only after the owned infraMachine is gone. This would give stronger guarantees that the underlying host is destroyed and so there's no chance of a stateful "leaked" process still running. kubernetes-sigs#2565
| @@ -261,7 +261,9 @@ func (r *MachineReconciler) reconcileDelete(ctx context.Context, cluster *cluste | |||
| logger := r.Log.WithValues("machine", m.Name, "namespace", m.Namespace) | |||
| logger = logger.WithValues("cluster", cluster.Name) | |||
|
|
|||
| if err := r.isDeleteNodeAllowed(ctx, m); err != nil { | |||
| err := r.isDeleteNodeAllowed(ctx, m) | |||
| isDeleteNodeAllowed := err == nil | |||
There was a problem hiding this comment.
this is a little weird to read, but I see why it was done this way. I'm ok keeping these changes and rewriting it later if we find a better way
|
LGTM |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/approve |
k8s-ci-robot
removed
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: enxebre, vincepri The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
/milestone v0.3.0 |
|
@vincepri: The provided milestone is not valid for this repository. Milestones in this repository: [ Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/milestone v0.3.1 |
What this PR does / why we need it:
If there were an edge case where we deleted an unreachable node and therefore removing any pod from etcd, but an stateful process were still running on the underlying host, we'll be immediately freeing up the pod name from the apiserver. "This would let the StatefulSet controller create a replacement Pod with that same identity; this can lead to the duplication of a still-running Pod, and if said Pod can still communicate with the other members of the StatefulSet, will violate the at most one semantics that StatefulSet is designed to guarantee."
To mitigate this it'd be safer to delete the node only after the owned infraMachine is gone. This would give stronger guarantees that the underlying host is destroyed and so there's no chance of a stateful "leaked" process still running.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #2565