NodeConformance testing with different OCI runtimes

[kannon92]

What would you like to be added?

We would like to add node conformance tests with crun.

To start, we would like to add presubmits and periodics for crun and crio.

We would like to switch runc to crun for the NodeConformance tests just to verify that there is no difference between these two OCI runtimes.

Why is this needed?

We want to make sure that there is no regressions between runc and crun.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kannon92]

/sig node

[kannon92]

/cc @mrunalp @haircommander

[BenTheElder]

We want to make sure that there is no regressions between runc and crun.

Shouldn't that be the responsibility of those projects?

We don't advertise support for runc/crun, and I would expect those projects to be running their own regression testing.

Are we running at HEAD? If not, what versions and why?

[kannon92]

I added this as an agenda item for sig-node for this Tuesday.

[BenTheElder]

Kubernetes does not make a habit of testing e.g., all CNI implementations, if we have a clearly defined API (like OCI) then compatibility testing should be pushed out into the implementation projects.

We use something to integration test Kubernetes with real clusters, but we don't advertise it or proclaim support for it, and those projects still need their own CI, and we certainly don't test the entire matrix, and the purpose of those jobs is almost exclusively to test Kubernetes, NOT the integrations we happen to use to get a cluster up.

We currently have a narrow exception in our CI hosting for containerd as a CNCF project we work with, and cadvisor as a necessary dependency, but we don't host CI for third party projects and I'm not sure we should be for containerd and cadvisor either.

We kicked out all other non-Kubernetes-project-repo CI in #12863 and do not test them with our resources, previously these resources were operated by Google and did include some additional projects.

Aside: can we please use the test-infra issue tracker for CI jobs etc? It is more targeted to that purpose.

[kannon92]

/transfer kubernetes/test-infra

[k8s-ci-robot]

@kannon92: Something went wrong or the destination repo kubernetes/kubernetes/test-infra does not exist.

In response to this:

/transfer kubernetes/test-infra

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kannon92]

/transfer test-infra

[kannon92]

Aside: can we please use the test-infra issue tracker for CI jobs etc? It is more targeted to that purpose.

I will do that for now on. Transferred the issue.

[BenTheElder]

Also: The issue is not with these projects, it's with the proliferation of "conformance test with ___" which is not sustainable and will have everyone vying to add their favorite configuration.

We generally do not overtly test third party integrations conformance or otherwise, sometimes it happens incidentally by necessity (e.g. most clusters are on GCP and AWS because that's where the funding is, so they use the necessary CSI drivers etc).

Aside: NodeConformance != "Conformance"

/retitle NodeConformance testing with different OCI runtimes

For the main Kubernetes conformance tests, we have a few jobs that run them, but we only mention that we're testing with e.g. "kind" or "kube-up" from within the project itself, to make sure the tests themselves work for use with other clusters. There are some other related jobs to e.g. ensure that the conformance tests do not depend on non-GA APIs by running a kind cluster that disables them.

What we do NOT do is publish / run "conformance with cilium" "conformance with containerd" "conformance with debian" "conformance with GKE"

[saschagrunert]

We would like to add node conformance tests with crun.

I'm wondering why we don't switch cgroupv2 completely to crun if the project decides to use that runtime as default in the future.

[samuelkarp]

I'm wondering why we don't switch cgroupv2 completely to crun if the project decides to use that runtime as default in the future.

I'm having trouble parsing this sentence. Does "the project" refer to Kubernetes or CRI-O? For context, containerd and runc both fully support cgroupv2 as well.

[kannon92]

I think sasha's suggestion was to switch the crio cgroupv2 job to use crun by default.

So our team is discussing our goal here. We were hoping that having tests for runc and crun could catch tricky integration bugs but it sounds like having replicated jobs for crun versus runc is not going to fly.

For context, we are exploring crun as our default for CRI-O and we discovered some edge cases in kubernetes around crun/runc that we did not catch in crun or CRI-O CI.

K8s PR: kubernetes/kubernetes#126641
crun PR: containers/crun#1517

[kannon92]

@BenTheElder @aojea

We met in sig-node to discuss this.

In CRI-O, we are going to explore ways to pull node-e2e into our CI so we can run the matrix that we want to support in the CRI-O side.

Our goal would be to confident to suggest that crun will be our default for CRI-O and once that happens, we will update our k8s CRI-O jobs to use crun instead of runc.

Since we are proposing release blocking jobs for CRI-O, we are thinking that we would keep those to be runc and create a counterpart for crun so not to block the release.

Timeframe on this is TBD as we need to flesh out the CRI-O CI first. cri-o/cri-o#8503

[kannon92]

/triage accepted

[kannon92]

/close

[k8s-ci-robot]

@kannon92: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.