kubevirt,vgpu: Bump vgpu lanes to use new kind-1.30-vgpu provider #3499

brianmcarey · 2024-06-28T13:30:21Z

What this PR does / why we need it:

The vgpu lanes are currently testing against kubernetes v1.27 which is no longer supported on the main branch of kubevirt.

Update these lanes to use the new provider kind-1.30-vgpu.

This requires the following PR to be merged into kubevirt first: kubevirt/kubevirt#12244

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

/cc @dhiller @xpivarc

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered

Release note:

brianmcarey · 2024-06-28T13:31:01Z

/hold

as mentioned in the description - this requires kubevirt/kubevirt#12244 to be merged.

brianmcarey · 2024-07-02T15:34:42Z

/rehearse

kubevirt-bot · 2024-07-02T15:35:39Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-07-04T07:20:57Z

/rehearse

kubevirt-bot · 2024-07-04T07:21:44Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-07-05T07:23:24Z

/rehearse

kubevirt-bot · 2024-07-05T07:23:29Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-07-15T08:00:13Z

/rehearse

kubevirt-bot · 2024-07-15T08:00:46Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

dhiller · 2024-07-16T08:16:10Z

@brianmcarey so I figure due to the still flaky history around the lane we are not there yet?

brianmcarey · 2024-07-26T07:27:20Z

/rehearse

kubevirt-bot · 2024-07-26T07:27:27Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-08-12T11:59:32Z

/rehearse

kubevirt-bot · 2024-08-12T11:59:38Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

xpivarc · 2024-08-13T12:47:10Z

@brianmcarey is this ready?

brianmcarey · 2024-08-13T12:56:18Z

@brianmcarey is this ready?

No still seeing intermittent failures on cluster-up. I haven't had a chance to come back to this yet.

brianmcarey · 2024-08-13T12:56:24Z

/rehearse

kubevirt-bot · 2024-08-13T12:56:32Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-08-29T14:09:46Z

/rehearse

kubevirt-bot · 2024-08-29T14:10:01Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

dhiller

/approve

@brianmcarey should we repeatedly run rehearse, so we get more feedback here, or do you think this is not required?

kubevirt-bot · 2024-08-29T15:31:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhiller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/OWNERS~~ [dhiller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

brianmcarey · 2024-08-29T16:38:22Z

/approve

@brianmcarey should we repeatedly run rehearse, so we get more feedback here, or do you think this is not required?

Yes I still think its not stable.

/rehearse

kubevirt-bot · 2024-08-29T16:38:27Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-09-04T08:40:53Z

/rehearse

kubevirt-bot · 2024-09-04T08:41:02Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

The vgpu lanes are currently testing against kubernetes v1.27 which is no longer supported on the main branch of kubevirt. Update these lanes to use the new provider kind-1.30-vgpu. This requires the following PR to be merged into kubevirt first: kubevirt/kubevirt#12244 Signed-off-by: Brian Carey <bcarey@redhat.com>

kubevirt-bot · 2024-09-12T13:12:17Z

New changes are detected. LGTM label has been removed.

brianmcarey · 2024-09-12T13:12:55Z

/rehearse

kubevirt-bot · 2024-09-12T13:13:03Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

brianmcarey · 2024-09-12T13:48:55Z

/rehearse

kubevirt-bot · 2024-09-12T13:49:02Z

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

kubevirt-bot · 2024-09-12T14:12:24Z

@brianmcarey: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-project-infra-prow-deploy-test	`959e381`	link	true	`/test pull-project-infra-prow-deploy-test`
rehearsal-pull-kubevirt-e2e-kind-1.30-vgpu	`959e381`	link	unknown	`/test pull-kubevirt-e2e-kind-1.30-vgpu`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

brianmcarey · 2024-09-17T13:06:30Z

I have been looking into the failures with this new provider - about 50% of the time the kind cluster fails to create due to some missing cgroups in the kind node container. I added a retry to the kind cluster create in the provider and this results in the provider coming up successfully every time. I haven't been able to identify why these cgroups are present sometimes and not present others - there maybe some issue with the mount that kind does on /sys

For now it is probably good enough to just add the retry in the provider.

Welcome to Debian GNU/Linux 12 (bookworm)!

Failed to attach 1 to compat systemd cgroup /init.scope: No such file or directory
Couldn't move remaining userspace processes, ignoring: Input/output error
Queued start job for default target graphical.target.
-.slice: Failed to migrate controller cgroups from /, ignoring: Input/output error
[  OK  ] Created slice kubelet.slic… used to run Kubernetes / Kubelet.
[  OK  ] Created slice system-modpr…lice - Slice /system/modprobe.
[  OK  ] Started systemd-ask-passwo…quests to Console Directory Watch.
[  OK  ] Set up automount proc-sys-…rmats File System Automount Point.
[  OK  ] Reached target cryptsetup.…get - Local Encrypted Volumes.
[  OK  ] Reached target integrityse…Local Integrity Protected Volumes.
[  OK  ] Reached target paths.target - Path Units.
[  OK  ] Reached target slices.target - Slice Units.
[  OK  ] Reached target swap.target - Swaps.
[  OK  ] Reached target veritysetup… - Local Verity Protected Volumes.
[  OK  ] Listening on systemd-journ…socket - Journal Audit Socket.
[  OK  ] Listening on systemd-journ…t - Journal Socket (/dev/log).
[  OK  ] Listening on systemd-journald.socket - Journal Socket.
[  OK  ] Reached target sockets.target - Socket Units.
Failed to attach 146 to compat systemd cgroup /dev-hugepages.mount: No such file or directory
         Mounting dev-hugepages.mount - Huge Pages File System...
Failed to attach 146 to compat systemd cgroup /dev-hugepages.mount: No such file or directory
Failed to attach 147 to compat systemd cgroup /sys-kernel-debug.mount: No such file or directory
         Mounting sys-kernel-debug.… - Kernel Debug File System...
Failed to attach 147 to compat systemd cgroup /sys-kernel-debug.mount: No such file or directory
Failed to attach 149 to compat systemd cgroup /sys-kernel-tracing.mount: No such file or directory
         Mounting sys-kernel-tracin… - Kernel Trace File System...
Failed to attach 149 to compat systemd cgroup /sys-kernel-tracing.mount: No such file or directory

xpivarc · 2024-09-17T13:46:21Z

@brianmcarey Have you seen the issues on cgroupv2?

brianmcarey · 2024-09-17T14:58:22Z

@brianmcarey Have you seen the issues on cgroupv2?

Trying locally with cgroups v2 we hit a different issue

Sep 17 14:57:16 vgpu-control-plane systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 17 14:57:16 vgpu-control-plane systemd[1]: Starting kubelet.service - kubelet: The Kubernetes Node Agent...
Sep 17 14:57:16 vgpu-control-plane sh[489]: ERROR: this script needs /sys/fs/cgroup/cgroup.procs to be empty (for writing the top-level cgroup.subtree_control)

https://github.com/kubernetes-sigs/kind/blob/52394ea8a92eed848d086318e983697f4a5afa93/images/base/files/kind/bin/create-kubelet-cgroup-v2.sh#L26

brianmcarey · 2024-09-18T08:36:02Z

With the following applied the lane passes every time:
https://github.com/kubevirt/kubevirtci/pull/1275/files

kubevirt-bot requested review from dhiller and xpivarc June 28, 2024 13:30

kubevirt-bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. size/XS labels Jun 28, 2024

kubevirt-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 28, 2024

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 12, 2024

brianmcarey force-pushed the bump-vgpu-1.30 branch from 3f5e03d to 3c59366 Compare July 15, 2024 08:00

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2024

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 13, 2024

brianmcarey force-pushed the bump-vgpu-1.30 branch from 3c59366 to 961afd7 Compare August 29, 2024 14:09

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 29, 2024

dhiller approved these changes Aug 29, 2024

View reviewed changes

kubevirt-bot assigned dhiller Aug 29, 2024

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 29, 2024

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 29, 2024

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 11, 2024

brianmcarey force-pushed the bump-vgpu-1.30 branch from 961afd7 to 959e381 Compare September 12, 2024 13:12

kubevirt-bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 12, 2024

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 12, 2024

brianmcarey mentioned this pull request Sep 19, 2024

kind-1.30-vgpu: Add retry to the creation of the kind cluster kubevirt/kubevirtci#1275

Open

8 tasks

kubevirt,vgpu: Bump vgpu lanes to use new kind-1.30-vgpu provider #3499

Are you sure you want to change the base?

kubevirt,vgpu: Bump vgpu lanes to use new kind-1.30-vgpu provider #3499

Conversation

brianmcarey commented Jun 28, 2024

brianmcarey commented Jun 28, 2024

brianmcarey commented Jul 2, 2024

kubevirt-bot commented Jul 2, 2024

brianmcarey commented Jul 4, 2024

kubevirt-bot commented Jul 4, 2024

brianmcarey commented Jul 5, 2024

kubevirt-bot commented Jul 5, 2024

brianmcarey commented Jul 15, 2024

kubevirt-bot commented Jul 15, 2024

dhiller commented Jul 16, 2024

brianmcarey commented Jul 26, 2024

kubevirt-bot commented Jul 26, 2024

brianmcarey commented Aug 12, 2024

kubevirt-bot commented Aug 12, 2024

xpivarc commented Aug 13, 2024

brianmcarey commented Aug 13, 2024

brianmcarey commented Aug 13, 2024

kubevirt-bot commented Aug 13, 2024

brianmcarey commented Aug 29, 2024

kubevirt-bot commented Aug 29, 2024

dhiller left a comment

Choose a reason for hiding this comment

kubevirt-bot commented Aug 29, 2024

brianmcarey commented Aug 29, 2024

kubevirt-bot commented Aug 29, 2024

brianmcarey commented Sep 4, 2024

kubevirt-bot commented Sep 4, 2024

kubevirt-bot commented Sep 12, 2024

brianmcarey commented Sep 12, 2024

kubevirt-bot commented Sep 12, 2024

brianmcarey commented Sep 12, 2024

kubevirt-bot commented Sep 12, 2024

kubevirt-bot commented Sep 12, 2024

brianmcarey commented Sep 17, 2024 • edited Loading

xpivarc commented Sep 17, 2024

brianmcarey commented Sep 17, 2024 • edited Loading

brianmcarey commented Sep 18, 2024

brianmcarey commented Sep 17, 2024 •

edited

Loading

brianmcarey commented Sep 17, 2024 •

edited

Loading