Propagate kubeadm dual-stack feature-gate to all k8s components #80531

Arvinderpal · 2019-07-24T15:05:32Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

This PR propagates the kubeadm feature-gate, if set in ClusterConfiguration, to the all individual kubernetes components on the control-plane node.

kind: ClusterConfiguration
featureGates:
  IPv6DualStack: true

For worker nodes, users will have to specify the dual-stack feature-gate to kubelet using the nodeRegistration.kubeletExtraArgs option as part of their join config. Alternatively, they can use KUBELET_EXTRA_ARGS as well.

Which issue(s) this PR fixes:

This addresses one of the TODO items in the dual-stack kubeadm tracker ticket: kubernetes/kubeadm#1612
It's a follow up to the initial kubeadm dual-stack feature-gate PR: #80145

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

In order to enable dual-stack support within kubeadm and kubernetes components, as part of the init config file, the user should set feature-gate IPv6DualStack=true in the ClusterConfiguration. Additionally, for each worker node, the user should set the feature-gate for kubelet using either nodeRegistration.kubeletExtraArgs or  KUBELET_EXTRA_ARGS.

k8s-ci-robot · 2019-07-24T15:05:39Z

Hi @Arvinderpal. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Arvinderpal · 2019-07-24T15:09:28Z

@neolit123 @yastij @fabriziopandini @aojea
As requested, this PR propagates the dual-stack feature-gate flag to all kubernetes components.
I made this a separate PR from the comma separated list support for --cluster-cidr. I did not want to mix these two together.

neolit123 · 2019-07-24T16:05:56Z

/assign
/priority important-longterm
/ok-to-test

neolit123

thanks @Arvinderpal
added minor comments, but overall this is looking good.

cmd/kubeadm/app/util/config/common.go

neolit123 · 2019-07-24T22:53:04Z

cmd/kubeadm/app/util/config/initconfiguration.go

@@ -102,6 +105,11 @@ func SetNodeRegistrationDynamicDefaults(cfg *kubeadmapi.NodeRegistrationOptions,
 		klog.V(1).Infof("detected and using CRI socket: %s", cfg.CRISocket)
 	}

+	if isDualStack {


can be:
if features.Enabled(cfg.ClusterConfiguration.FeatureGates, features.IPv6DualStack) {
to avoid passing the isDualStack arg
[1]

That won't work since we pass in cfg *kubeadmapi.NodeRegistrationOptions and not ClusterConfiguration

ah i see, so one solution is to make SetNodeRegistrationDynamicDefaults accept ClusterConfiguration (it think it used to be like that at some point) 🤔

@rosti WDYT?
/cc @rosti

This is not the place to do the feature gate patching for the kubelet. The correct place is buildKubeletArgMap:

kubernetes/cmd/kubeadm/app/phases/kubelet/flags.go

Line 78 in fc4cbf7

func buildKubeletArgMap(opts kubeletFlagsOpts) map[string]string {

@rosti I see a problem with how BuildArgumentListFromMap works today -- it does not handle comma separated fields like "feature-gates", since it does a simple overwrite of default values with user specified values. For example, if we set IPv6DualStack=true in base, then that will be overwritten if the user has specified some additional feature-gates in KubeletExtraArgs.

We have a couple of options:

Make BuildArgumentListFromMap capable of "merging" fields like feature-gates rather than simple key/value overwrite. We would do something like this:

for k, v := range overrideArguments { if k == "feature-gates" && len(argsMap[k]) > 0{ argsMap[k] += "," + v }else{ argsMap[k] = v } }

Modify top-level funcs like WriteKubeletDynamicEnvFile so that they perform the merge on feature-gates. This would require updating the argList list returned by BuildArgumentListFromMap. This is not ideal, IMO.

We could also just leave the code in the higher-level funcs as it is in the current PR.
I'll wait for you feedback before proceeding. Thanks.

Let's keep things simple for now and just allow the users to overwrite the whole of the feature-gates flags if they wish.
Either way, this is going to be documented as part of the feature gate documentation.
Another possibility is to modify BuildArgumentListFromMap to warn on overwrites.

Certainly, we should not do this in the dynamic defaulting code.

@fabriziopandini @neolit123 WDYT?

i think that @rosti is correct here that dynamic defaults is not the best place.

Let's keep things simple for now and just allow the users to overwrite the whole of the feature-gates flags if they wish.

makes sense to me.

passing the dual stack feature gate to the kubelet will be implicit (if the kubeadm FG is on). so if the user provides their own feature-gates for the kubelet extraArgs, this should overwrite the implicit gate but should also show a warning.

Either way, this is going to be documented as part of the feature gate documentation.

we might add a sentence about overrides, yes.

cmd/kubeadm/app/util/config/initconfiguration.go

cmd/kubeadm/app/util/config/common.go

cmd/kubeadm/app/util/config/common_test.go

neolit123 · 2019-07-24T23:04:51Z

/retest

Arvinderpal

@neolit123 Thanks for the feedback. Please see my comments.

Arvinderpal · 2019-07-25T00:20:10Z

cmd/kubeadm/app/util/config/initconfiguration.go

@@ -102,6 +105,11 @@ func SetNodeRegistrationDynamicDefaults(cfg *kubeadmapi.NodeRegistrationOptions,
 		klog.V(1).Infof("detected and using CRI socket: %s", cfg.CRISocket)
 	}

+	if isDualStack {


That won't work since we pass in cfg *kubeadmapi.NodeRegistrationOptions and not ClusterConfiguration

cmd/kubeadm/app/util/config/initconfiguration.go

cmd/kubeadm/app/util/config/common.go

cmd/kubeadm/app/util/config/common_test.go

aojea · 2019-07-25T20:07:14Z

/test pull-kubernetes-integration

rosti

Thanks @Arvinderpal !
However, we need the feature gate propagation to be done at a more lower code level, than in the config dynamic defaulting code.

rosti · 2019-07-26T12:22:24Z

cmd/kubeadm/app/util/config/initconfiguration.go

@@ -102,6 +105,11 @@ func SetNodeRegistrationDynamicDefaults(cfg *kubeadmapi.NodeRegistrationOptions,
 		klog.V(1).Infof("detected and using CRI socket: %s", cfg.CRISocket)
 	}

+	if isDualStack {


This is not the place to do the feature gate patching for the kubelet. The correct place is buildKubeletArgMap:

kubernetes/cmd/kubeadm/app/phases/kubelet/flags.go

Line 78 in fc4cbf7

func buildKubeletArgMap(opts kubeletFlagsOpts) map[string]string {

cmd/kubeadm/app/util/config/initconfiguration.go

Arvinderpal · 2019-07-29T15:43:20Z

@rosti @neolit123 Thank you for the feedback. I have made the changes requested.
It actually makes things a lot simpler now :)

neolit123

thanks for the update
/lgtm
approve on @rosti

neolit123 · 2019-07-29T15:55:04Z

looks like some build tests are failing.

yastij

just some nits, also +1 on the previous feedback @rosti

yastij · 2019-07-30T08:03:39Z

cmd/kubeadm/app/phases/controlplane/manifests.go

+	enabled, present := cfg.FeatureGates[features.IPv6DualStack]
+	if present && enabled {
+		defaultArguments["feature-gates"] = features.IPv6DualStack + "=true"
+	} else if present && !enabled {


why not just else ? (the remaining case is !present)

[1] Ideally, we would like to cover the present==true case. So something like this should work:

if present { defaultArguments["feature-gates"] = fmt.Sprintf("%s=%t", features.IPv6DualStack, enabled) }

cmd/kubeadm/app/phases/controlplane/manifests.go

cmd/kubeadm/app/phases/kubelet/flags.go

rosti

Thanks @Arvinderpal !
Leaving final lgtm to @yastij or @neolit123
/approve

rosti · 2019-07-30T08:32:23Z

cmd/kubeadm/app/phases/controlplane/manifests.go

+	enabled, present := cfg.FeatureGates[features.IPv6DualStack]
+	if present && enabled {
+		defaultArguments["feature-gates"] = features.IPv6DualStack + "=true"
+	} else if present && !enabled {


[1] Ideally, we would like to cover the present==true case. So something like this should work:

if present { defaultArguments["feature-gates"] = fmt.Sprintf("%s=%t", features.IPv6DualStack, enabled) }

cmd/kubeadm/app/phases/controlplane/manifests.go

cmd/kubeadm/app/phases/kubelet/flags.go

k8s-ci-robot · 2019-07-30T08:36:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Arvinderpal, rosti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [rosti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

progagates the kubeadm FG to the individual k8scomponents on the control-plane node. * Note: Users who want to join worker nodes to the cluster will have to specify the dual-stack FG to kubelet using the nodeRegistration.kubeletExtraArgs option as part of their join config. Alternatively, they can use KUBELET_EXTRA_ARGS. kubeadm FG: kubernetes/kubeadm#1612

yastij · 2019-07-30T14:34:21Z

/lgtm

Arvinderpal · 2019-07-30T16:48:43Z

/retest

fejta-bot · 2019-07-30T21:58:09Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

fejta-bot · 2019-07-31T00:46:33Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

fejta-bot · 2019-07-31T07:04:33Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

k8s-ci-robot · 2019-07-31T07:26:29Z

@Arvinderpal: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce	`585ef37`	link	`/test pull-kubernetes-e2e-gce`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot requested review from dixudx and fabriziopandini July 24, 2019 15:07

Arvinderpal mentioned this pull request Jul 24, 2019

Dual-Stack support for Kubeadm kubernetes/kubeadm#1612

Closed

14 tasks

k8s-ci-robot assigned neolit123 Jul 24, 2019

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch 2 times, most recently from 0556136 to 86be728 Compare July 24, 2019 22:33

neolit123 reviewed Jul 24, 2019

View reviewed changes

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from 86be728 to ec15c65 Compare July 25, 2019 00:53

Arvinderpal commented Jul 25, 2019

View reviewed changes

k8s-ci-robot requested a review from rosti July 25, 2019 02:08

rosti suggested changes Jul 26, 2019

View reviewed changes

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from ec15c65 to 56ad9b6 Compare July 29, 2019 15:38

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 29, 2019

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from 56ad9b6 to 6a58483 Compare July 29, 2019 15:40

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from 6a58483 to 6082e7f Compare July 29, 2019 15:45

neolit123 reviewed Jul 29, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 29, 2019

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from 6082e7f to bec0017 Compare July 30, 2019 04:14

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 30, 2019

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from bec0017 to 6524f81 Compare July 30, 2019 05:18

yastij reviewed Jul 30, 2019

View reviewed changes

rosti reviewed Jul 30, 2019

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 30, 2019

Arvinderpal force-pushed the kubeadm-ds-FG-propagate branch from 6524f81 to 585ef37 Compare July 30, 2019 14:28

k8s-ci-robot assigned yastij Jul 30, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 30, 2019

k8s-ci-robot merged commit 5bfa366 into kubernetes:master Jul 31, 2019

k8s-ci-robot added this to the v1.16 milestone Jul 31, 2019

Arvinderpal mentioned this pull request Nov 27, 2019

REQUEST: New membership for Arvinderpal kubernetes/org#1443

Closed

6 tasks

aojea mentioned this pull request Jan 3, 2020

DualStack: Fail to deploy dualstack cluster, kube-proxy panics #86773

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propagate kubeadm dual-stack feature-gate to all k8s components #80531

Propagate kubeadm dual-stack feature-gate to all k8s components #80531

Arvinderpal commented Jul 24, 2019

k8s-ci-robot commented Jul 24, 2019

Arvinderpal commented Jul 24, 2019

neolit123 commented Jul 24, 2019

neolit123 left a comment

neolit123 Jul 24, 2019

Arvinderpal Jul 25, 2019

neolit123 Jul 25, 2019

rosti Jul 26, 2019

Arvinderpal Jul 28, 2019

rosti Jul 29, 2019

neolit123 Jul 29, 2019

Arvinderpal Jul 29, 2019

neolit123 commented Jul 24, 2019

Arvinderpal left a comment

Arvinderpal Jul 25, 2019

aojea commented Jul 25, 2019

rosti left a comment

rosti Jul 26, 2019

Arvinderpal commented Jul 29, 2019

neolit123 left a comment

neolit123 commented Jul 29, 2019

yastij left a comment

yastij Jul 30, 2019

rosti Jul 30, 2019

Arvinderpal Jul 30, 2019

rosti left a comment

rosti Jul 30, 2019

k8s-ci-robot commented Jul 30, 2019

yastij commented Jul 30, 2019

Arvinderpal commented Jul 30, 2019

fejta-bot commented Jul 30, 2019

fejta-bot commented Jul 31, 2019

fejta-bot commented Jul 31, 2019

k8s-ci-robot commented Jul 31, 2019

Propagate kubeadm dual-stack feature-gate to all k8s components #80531

Propagate kubeadm dual-stack feature-gate to all k8s components #80531

Conversation

Arvinderpal commented Jul 24, 2019

k8s-ci-robot commented Jul 24, 2019

Arvinderpal commented Jul 24, 2019

neolit123 commented Jul 24, 2019

neolit123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 commented Jul 24, 2019

Arvinderpal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Jul 25, 2019

rosti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arvinderpal commented Jul 29, 2019

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 commented Jul 29, 2019

yastij left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 30, 2019

yastij commented Jul 30, 2019

Arvinderpal commented Jul 30, 2019

fejta-bot commented Jul 30, 2019

fejta-bot commented Jul 31, 2019

fejta-bot commented Jul 31, 2019

k8s-ci-robot commented Jul 31, 2019