Allow Hostname and Subdomain to be set if empty #51199

kow3ns · 2017-08-23T15:30:26Z

What this PR does / why we need it:
This PR allows the Hostname and Subdomain field of v1.PodSpec to be set when empty, and modifies the StatefulSet controller to set them when empty.

For #48327:
We have merged #50942 to ensure that the Hostname and Subdomain fields are set when a new Pod is created. Users should upgrade to 1.6.9 and perform a rolling restart of all Pods in their StatefulSets to ensure that these fields are set prior to an upgrade to 1.7.5.
We have merged #51149 and #51044 to rollback the attempted mutation introduced in #44137.
This PR allows the Hostname and Subdomain field to be set exactly once, so that when users fail to read the notes, and encounter this issue, their clusters should self heal (even though they will experience a temporary network disruption for Pods in their StatefulSets.)

StatefulSet will now fill the `hostname` and `subdomain` fields if they're empty on existing Pods it owns. This allows it to self-correct the issue where StatefulSet Pod DNS entries disappear after upgrading to v1.7.x (#48327).

k8s-cherrypick-bot · 2017-08-23T15:33:46Z

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

enisoc · 2017-08-23T17:31:47Z

pkg/api/validation/validation_test.go

+				Spec:       api.PodSpec{Hostname: "bar"},
+			},
+			api.Pod{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", DeletionTimestamp: &now},


Why the DeletionTimestamp?

enisoc · 2017-08-23T17:41:28Z

You verified that endpoints will react to the Pod update and bring back the DNS entry?

enisoc · 2017-08-23T17:43:55Z

It looks like we will never reach updateIdentity() unless the Pod is Ready:

kubernetes/pkg/controller/statefulset/stateful_set_control.go

Lines 420 to 443 in 2f00e6d

    
           // If we have a Pod that has been created but is not running and ready we can not make progress. 
        
           // We must ensure that all for each Pod, when we create it, all of its predecessors, with respect to its 
        
           // ordinal, are Running and Ready. 
        
           if !isRunningAndReady(replicas[i]) && monotonic { 
        
           	glog.V(4).Infof( 
        
           		"StatefulSet %s/%s is waiting for Pod %s to be Running and Ready", 
        
           		set.Namespace, 
        
           		set.Name, 
        
           		replicas[i].Name) 
        
           	return &status, nil 
        
           } 
        
           // Enforce the StatefulSet invariants 
        
           if identityMatches(set, replicas[i]) && storageMatches(set, replicas[i]) { 
        
           	continue 
        
           } 
        
           // Make a deep copy so we don't mutate the shared cache 
        
           copy, err := scheme.Scheme.DeepCopy(replicas[i]) 
        
           if err != nil { 
        
           	return &status, err 
        
           } 
        
           replica := copy.(*v1.Pod) 
        
           if err := ssc.podControl.UpdateStatefulPod(updateSet, replica); err != nil { 
        
           	return &status, err 
        
           }

Is it worth handling the case where the Pod is Unready precisely because its hostname/subdomain is missing?

kow3ns · 2017-08-24T17:59:35Z

It may be worth doing the update, that is the way the control loop was originally written. If we do this, it should go into master and get cherry picked back. I'd like to minimize drift in the main control loop.

enisoc · 2017-08-25T17:47:56Z

/lgtm

@kow3ns Are you planning to address Unready Pods before 1.7.5 as well?

smarterclayton · 2017-09-01T16:21:25Z

pkg/controller/statefulset/stateful_set_utils.go

@@ -195,7 +195,12 @@ func initIdentity(set *apps.StatefulSet, pod *v1.Pod) {
 func updateIdentity(set *apps.StatefulSet, pod *v1.Pod) {
 	pod.Name = getPodName(set, getOrdinal(pod))
 	pod.Namespace = set.Namespace
-
+	if pod.Spec.Hostname == "" {
+		pod.Spec.Hostname = pod.Name


Why isn't this reading the annotation as well?

The annotations and the fields are both constructed from the name and serviceName as below. What information would I want from the existing annotations?

lavalamp · 2017-09-14T20:01:57Z

pkg/api/validation/validation.go

+
+	// allow hostname and subdomain to be updated if they are empty. This allows for migration between the beta
+	// annotations and the GA field
+	if oldPod.Spec.Hostname == "" && oldPod.Spec.Hostname != newPod.Spec.Hostname {


second clause on both of these if statements is not actually necessary...

lavalamp · 2017-09-14T20:03:21Z

pkg/api/validation/validation.go

@@ -2703,6 +2703,16 @@ func ValidatePodUpdate(newPod, oldPod *api.Pod) field.ErrorList {

 	// handle updateable fields by munging those fields prior to deep equal comparison.
 	mungedPod := *newPod
+
+	// allow hostname and subdomain to be updated if they are empty. This allows for migration between the beta
+	// annotations and the GA field


expand this comment, is it supposed to stay this way forever?

lavalamp · 2017-09-14T20:33:10Z

/approve

Need the branch manager to sign off, too.

enisoc · 2017-09-22T18:32:46Z

pkg/controller/statefulset/stateful_set_control.go

+		// Enforce the StatefulSet invariants - we do this without respect to the Pod's readiness so that the endpoints
+		// controller can be notified of identity changes if a Pod becomes unready due to a DNS inconsistency with respect
+		// to the Pods identity.
+		if !identityMatches(set, replicas[i]) || !storageMatches(set, replicas[i]) {


I tried a manual test of this and found a bug. We also need to put the check for hostname/subdomain back into identityMatches(), or else this doesn't trigger.

Automatic merge from submit-queue. StatefulSet: Fix auto-heal of Pod hostname/subdomain fields. This is a followup to #52557, which was a backport of #51199. The original fix was incomplete. It doesn't trigger unless we change the consistency check to look at the fields.

enisoc · 2017-09-22T23:17:22Z

pkg/controller/statefulset/stateful_set_utils.go

-		pod.Namespace == set.Namespace
+		pod.Namespace == set.Namespace &&
+		pod.Spec.Hostname != "" &&
+		pod.Spec.Subdomain != ""


I just had a thought:

ServiceName is allowed to be empty in 1.7+ [1], so we should do something like:

(pod.Spec.Subdomain != "" || set.ServiceName == "")

Otherwise we will continually send Pod updates for any StatefulSet with an empty ServiceName.

[1] The reason ServiceName can't be empty in <1.7 is because the annotation gets added anyway, and then the Pod fails validation. In 1.7+ we no longer set the annotation, and the field has no distinction between "" and unset.

and updates the StatefulSet controller to set them when empty

enisoc · 2017-09-25T22:43:33Z

/lgtm

k8s-github-robot · 2017-09-25T22:43:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enisoc, kow3ns, lavalamp

Associated issue: 48327

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~pkg/api/OWNERS~~ [lavalamp]
~~pkg/controller/statefulset/OWNERS~~ [enisoc,kow3ns,lavalamp]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

kow3ns · 2017-09-25T23:08:09Z

/retest

fejta-bot · 2017-09-26T03:51:25Z