containerd: use systemd cgroup driver by default?

[AkihiroSuda]

What would you like to be added:
Use systemd cgroup driver by default

Why is this needed:
Because it is recommended on systemd hosts: https://kubernetes.io/docs/setup/production-environment/container-runtimes/

[aojea]

this seems a simple change, @AkihiroSuda do you know there can be issues due to be running in nested containers?
I mean, let's says docker runs in the host with cgroupfs, and kubelet inside the kind node (docker container) with systemd driver (this systemd process is runnig inside a cgroup already)

[tao12345666333]

Maybe there is no problem?

Currently, my docker is using systemd as the cgroup driver, and cgroupfs is used in KIND.

I will test locally and give the results as soon as possible

[BenTheElder]

That page talks about having two controllers, but if it intends to recommend it, the language there could do with some updating, I wouldn't really call that "recommends". Nowhere does it say "you should do this".

I'd want to know that we're not going to cause a regression here.

We also want to be doing something like the changes in https://d2iq.com/blog/running-kind-inside-a-kubernetes-cluster-for-continuous-integration with custom kubepods path etc. sooner rather than later, cc @jieyu xref: #1614.

[AkihiroSuda]

For cgroup v2, runc explicitly recommends systemd driver, though it is still opt-in.
https://github.com/opencontainers/runc/blob/master/docs/cgroup-v2.md#systemd

Podman and Docker already switched the default on v2 to systemd.

[BenTheElder]

cc @neolit123
/assign

[BenTheElder]

We should strongly consider this in v0.11.0, see also kubernetes/kubernetes#96594 for some excitement around kind's (poor) behavior with cgroups v1 currently ...

[BenTheElder]

Noting that there's currently some issues upstream with runc upgrade and systemd
kubernetes/kubernetes#88197 kubernetes/kubernetes#102094

[BenTheElder]

There is some cri-o CI in Kubernetes related to this now but still this driver continues to have problems kubernetes/kubernetes#102508 (comment)

[BenTheElder]

This driver is getting more testing upstream but still seems to have major issues kubernetes/kubernetes#104280 😬

[adisky]

Now there is systemd cgroup job for containerd as well
https://testgrid.k8s.io/sig-node-containerd#cgroup-systemd-containerd-node-e2e

[BenTheElder]

I was just talking to @stevekuznetsov about this, we are in a weird state right now where we are cgroupfs + systemd.

kubeadm 1.22+ switched to systemd by default, and I think this is fairly well tested now. Also it is likely problematic on cgroupv2 (though ... kind is working there at least in CI environment and some users ...).

We should update kind build node-image to patch the default containerd config to use systemd cgroup driver for kubernetes 1.22+ builds.

That will fix this even for older kind binaries, and it's cheaper than another runtime check. It will still be possible to override at runtime with config patches.

[BenTheElder]

P.S. apologies in advance if anyone takes that up, that code is a bit messy in particular 😬 , it should not be a difficult change though.

[BenTheElder]

/remove-help
/assign

Existing proposal is not quite enough if we want to avoid breaking changes, because in the kubeadm config we emit from existing kind binaries we're specifying the kubelet cgroup driver (to avoid kubeadm in 1.21+ overriding the kubelet's actual default).

Considering alternatives. Will PR soon.

[BenTheElder]

In #2737, thinking is to basically do what kubeadm did, and just say "1.24.0 onwards uses systemd" (and will require the next kind release to use), treating it as breaking then (instead of 1.21 which we already patched over.

it's not unusual for a new kubernetes release to require a new kind version because of some change in kubernetes. the fact that we deferred it a few releases seems fine.

[BenTheElder]

kubernetes v1.24.0+ and kind v0.13.0+ will do this.