convert some jobs to Ginkgo --label-filter

[carlory]

Why we need this change:

• convert some jobs to Ginkgo --label-filter #32648
• WIP: remove unneeded e2e features kubernetes#125452
• Ginkgo Proposal - Label Sets

How to do it

• Some tips for figuring out how to replace --focus/skip with --label-filter:

  • go test -v ./test/e2e -args -list-tests (same for e2e_node) shows all tests.
  • Note that not all tags in [] are also labels. Only those listed by go test -v ./test/e2e -args -list-labels are proper labels. If you find that you need to filter by something that isn't a label yet, then replace the inline text with WithLabel.
  • _output/bin/ginkgo --dry-run [--focus/skip/label-filter=...] ./test/e2e can be used to compare how many and, when adding -v, which tests would run. Note that statistics for actual runs may differ when tests ask to be skipped when invoked. (install ginkgo via make ginkgo)

• If a job has a "canary" version, update that version first and try it out.

• When converting some real jobs:

  • Create a separate tracking issue with a link to the jobs' testgrid entry.
  • Convert the job, linking to that issue with Related-to: ... (not Fixes: ...!).
  • Verify that the job still works as intended, then close the issue.

• After converting a pre-submit job successfully, do the same for the corresponding periodic job.

• Some best practices:

  • Replace focus/skip completely with a single label-filter.
  • Include !Flaky even if you currently don't have flaky tests. Having it in the jobs is convenient when some tests turns out to be problematic and needs to be disable temporarily.
  • Exclude tests with special requirements with Feature: isEmpty.
  • Feature: containsAny <my feature> && Feature: isSubsetOf <my feature> runs tests with have the <my feature> label and no other unknown features. Use this in dedicated jobs for that feature. Note that e2e_node uses NodeFeature for the same purpose.
  • Ginkgo can run serial jobs together with parallel ones in a single invocation by running the serial jobs (and only those) sequentially, so it's not absolutely required to use !Serial when running ginkgo -p and there is no need for separate serial/parallel jobs.
  • Beware that serial tests can affect the overall runtime. It might be useful to skip with !Slow and !Serial in pre-submits and only run the skipped tests in periodic jobs.

What should be cleaned up or changed:

• sig-api-machinery
• sig-apps
• sig-arch
• sig-auth
• sig-autoscaling
• sig-cli
• sig-cloud-provider
• sig-cluster-lifecycle
• sig-instrumentation
• sig-k8s-infra
• sig-network
• sig-node
• sig-release
• sig-scalability
• sig-scheduling
• sig-security
• [SIG Storage] Convert all jobs to Ginkgo --label-filter  #33152 @carlory
• sig-testing
  • sig-testing: *-canary job uses LABEL_FILTER instead of FOCUS #32898 (we also need to clear skip)
  • sig-testing: uses LABEL_FILTER instead of FOCUS #32867 (but reverted by Revert "sig-testing: uses LABEL_FILTER instead of FOCUS" #33131 due to 🚨 all test cases being skipped in pull-kubernetes-e2e-kind 🚨  kubernetes#126401)
• sig-windows

[carlory]

/sig testing

[carlory]

/assign
cc @pohly @aojea

[aojea]

/cc @BenTheElder

[aojea]

@carlory it will be super useful to write some notes on how to perform the migration, so we document it and achieve some consistency to avoid different groups to end with different solutions to the same problem

[pohly]

Here's a proposal. @carlory: if this looks good, copy it into the description to ensure that it remains visible?

• Some tips for figuring out how to replace --focus/skip with --label-filter:
  • go test -v ./test/e2e -args -list-tests (same for e2e_node) shows all tests.
  • Note that not all tags in [] are also labels. Only those listed by go test -v ./test/e2e -args -list-labels are proper labels. If you find that you need to filter by something that isn't a label yet, then replace the inline text with WithLabel.
  • _output/bin/ginkgo --dry-run [--focus/skip/label-filter=...] ./test/e2e can be used to compare how many and, when adding -v, which tests would run. Note that statistics for actual runs may differ when tests ask to be skipped when invoked.
• If a job has a "canary" version, update that version first and try it out.
• When converting some real jobs:
  • Create a separate tracking issue with a link to the jobs' testgrid entry.
  • Convert the job, linking to that issue with Related-to: ... (not Fixes: ...!).
  • Verify that the job still works as intended, then close the issue.
• After converting a pre-submit job successfully, do the same for the corresponding periodic job.
• Some best practices:
  • Replace focus/skip completely with a single label-filter.
  • Include !Flaky even if you currently don't have flaky tests. Having it in the jobs is convenient when some tests turns out to be problematic and needs to be disable temporarily.
  • Exclude tests with special requirements with Feature: isEmpty.
  • Feature: containsAny <my feature> && Feature: isSubsetOf <my feature> runs tests with have the <my feature> label and no other unknown features. Use this in dedicated jobs for that feature. Note that e2e_node uses NodeFeature for the same purpose.
  • Ginkgo can run serial jobs together with parallel ones in a single invocation by running the serial jobs (and only those) sequentially, so it's not absolutely required to use !Serial when running ginkgo -p and there is no need for separate serial/parallel jobs.
  • Beware that serial tests can affect the overall runtime. It might be useful to skip with !Slow and !Serial in pre-submits and only run the skipped tests in periodic jobs.