Allow optionally setting node taints defined on the NodeFeatureRule CR #910

fmuyassarov · 2022-10-06T11:51:06Z

This PR enables setting node taints defined on NodeFeatureRule CR. User can
define a list of taints under spec.rule of the CR as in this example:

apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: rule
spec:
  rules:
    - name: "rule"
      labels:
        "my-sample-feature": "true"
      taints:
        - effect: PreferNoSchedule
          key: "feature.node.kubernetes.io/special-node"
          value: "true"
        - effect: NoExecute
          key: "feature.node.kubernetes.io/dedicated-node"
      matchFeatures:
        - feature: kernel.loadedmodule
          matchExpressions:
            dummy: {op: Exists}

$ kubectl get no kind-worker -ojson | jq .spec.taints
[
  {
    "effect": "PreferNoSchedule",
    "key": "feature.node.kubernetes.io/special-node",
    "value": "true"
  },
  {
    "effect": "NoExecute",
    "key": "feature.node.kubernetes.io/dedicated-node",
  }
]
$ kubectl get no kind-worker -ojson | jq .metadata.annotations | grep taint
  "nfd.node.kubernetes.io/taints": "feature.node.kubernetes.io/special-node=true:PreferNoSchedule,feature.node.kubernetes.io/dedicated-node:NoExecute",

The only allowed values for the effect are NoExecute, NoSchedule, PrefereNoSchedule
(following k8s). By default this feature is disabled and enabling it requires explicitly setting
--enable-taints flag true. Even if there are taints defined in the NodeFeatureRule CR and the
rule matches, taints will not be set on the node until flag is enabled.

NFD master updates the node object based on the addition/removal of taints in the CR.
Deleting the CR results in deletion of the taints too. nfd.node.kubernetes.io/taints annotation
is added to keep track of taints. The value of the new annotation is a list of taints in the format
of nfd.node.kubernetes.io/taints: "<key>=<value>:<effect>,<key>=<value>:<effect>".

Fixes: #540

netlify · 2022-10-06T11:51:29Z

✅ Deploy Preview for kubernetes-sigs-nfd ready!

Name	Link
🔨 Latest commit	`984a3de`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-nfd/deploys/638a19df59623f0008b6cd96
😎 Deploy Preview	https://deploy-preview-910--kubernetes-sigs-nfd.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

fmuyassarov · 2022-10-06T11:57:18Z

Ops, seems I'm behind the master. I'll rebase. Actually I missed to the part where taints are deleted based on the deletion of the NodeFeatureRule CR or the specific taints on the CR (will add it shortly).

fmuyassarov · 2022-10-06T12:34:30Z

/hold

fmuyassarov · 2022-10-08T23:08:37Z

/cc @marquiz
I think it is ready for the review and meanwhile I will add a commit with unit tests.

fmuyassarov · 2022-10-08T23:08:48Z

/hold cancel

fmuyassarov · 2022-10-09T11:38:10Z

/test pull-node-feature-discovery-build-image-cross-generic

fmuyassarov · 2022-10-10T14:15:13Z

/cc @ArangoGutierrez @zvonkok

marquiz

Thanks @fmuyassarov! I took a first quick pass (incomplete, not even going through all the changes, yet) and this looks really good already.

One suggestion/ask: could you submit the package renames

- whatever "k8s.io/api/core/v1"
+ corev1 "k8s.io/api/core/v1"

and other possible "beautifying" as separate PR(s). This will greatly reduce the "noise" in this PR and make review easier

pkg/apihelper/apihelpers.go

docs/advanced/customization-guide.md

pkg/apis/nfd/v1alpha1/annotations_labels.go

docs/advanced/customization-guide.md

pkg/apis/nfd/v1alpha1/expression.go

pkg/nfd-master/nfd-master-internal_test.go

marquiz · 2022-10-13T10:51:27Z

BTW, now that #848 is in, we should also add some e2e coverage. But that can be done in a separate PR (and tracked in a separate issue), imo

fmuyassarov · 2022-10-13T11:18:02Z

BTW, now that #848 is in, we should also add some e2e coverage. But that can be done in a separate PR (and tracked in a separate issue), imo

Yep, I'm on it now 😊

pkg/nfd-master/nfd-master.go

Extend current E2E tests to check tainting feature of nfd implemented in kubernetes-sigs#910 Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

Extend NodeFeatureRule Spec with taints field to allow users to specify the list of the taints they want to be set on the node if rule matches. Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

This commits extends NFD master code to support adding node taints from NodeFeatureRule CR. We also introduce a new annotation for taints which helps to identify if the taint set on node is owned by NFD or not. When user deletes the taint entry from NodeFeatureRule CR, NFD will remove the taint from the node. But to avoid accidental deletion of taints not owned by the NFD, it needs to know the owner. Keeping track of NFD set taints in the annotation can be used during the filtering of the owner. Also enable-taints flag is added to allow users opt in/out for node tainting feature. The flag takes precedence over taints defined in NodeFeatureRule CR. In other words, if enbale-taints is set to false(disabled) and user still defines taints on the CR, NFD will ignore those taints and skip them from setting on the node. Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

marquiz · 2022-12-02T15:29:02Z

@fmuyassarov please rebase
ping @ArangoGutierrez

Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

fmuyassarov · 2022-12-02T15:29:49Z

@fmuyassarov please rebase ping @ArangoGutierrez

done

marquiz

Thx @fmuyassarov

ArangoGutierrez

/lgtm

k8s-ci-robot · 2022-12-06T15:10:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ArangoGutierrez, fmuyassarov, marquiz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [marquiz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Extend current E2E tests to check tainting feature of nfd implemented in kubernetes-sigs#910 Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

This commit adds an argument to updateNodeFeatures method for receiving client argument, which currently gets initialized within the method itself. This is a minor improvement for kubernetes-sigs/node-feature-discovery#910. Ref:kubernetes-sigs/node-feature-discovery#910 (comment) Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

Extend current E2E tests to check tainting feature of nfd implemented in kubernetes-sigs/node-feature-discovery#910 Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

Extend current E2E tests to check tainting feature of nfd implemented in kubernetes-sigs#910 Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

k8s-ci-robot requested review from adrianchiris and ArangoGutierrez October 6, 2022 11:51

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 6, 2022

fmuyassarov force-pushed the taint/feruz branch from 20e44ec to 069ebee Compare October 6, 2022 12:27

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 6, 2022

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 6, 2022

fmuyassarov force-pushed the taint/feruz branch 2 times, most recently from 2c6ed87 to f100b61 Compare October 8, 2022 22:56

k8s-ci-robot requested a review from marquiz October 8, 2022 23:08

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 8, 2022

fmuyassarov force-pushed the taint/feruz branch from f100b61 to 9e42d04 Compare October 9, 2022 21:34

k8s-ci-robot requested a review from zvonkok October 10, 2022 14:15

marquiz reviewed Oct 13, 2022

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 13, 2022

marquiz reviewed Oct 13, 2022

View reviewed changes

pkg/nfd-master/nfd-master.go Outdated Show resolved Hide resolved

fmuyassarov force-pushed the taint/feruz branch from 9e42d04 to a990e1b Compare October 14, 2022 08:55

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 14, 2022

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 2, 2022

fmuyassarov added 2 commits December 2, 2022 17:25

Add taints field to NodeFeatureRule CR spec

532e119

Extend NodeFeatureRule Spec with taints field to allow users to specify the list of the taints they want to be set on the node if rule matches. Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

Document tainting feature

984a3de

Signed-off-by: Feruzjon Muyassarov <feruzjon.muyassarov@intel.com>

fmuyassarov force-pushed the taint/feruz branch from 6fcc0f0 to 984a3de Compare December 2, 2022 15:29

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 2, 2022

marquiz approved these changes Dec 2, 2022

View reviewed changes

ArangoGutierrez approved these changes Dec 6, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 6, 2022

k8s-ci-robot merged commit 9f68f6c into kubernetes-sigs:master Dec 6, 2022

fmuyassarov deleted the taint/feruz branch December 7, 2022 12:45

marquiz mentioned this pull request Dec 20, 2022

Release v0.12.0 #985

Closed

22 tasks

marquiz mentioned this pull request Mar 11, 2023

Discover node features as annotations #863

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow optionally setting node taints defined on the NodeFeatureRule CR #910

Allow optionally setting node taints defined on the NodeFeatureRule CR #910

fmuyassarov commented Oct 6, 2022 •

edited

Loading

netlify bot commented Oct 6, 2022 •

edited

Loading

fmuyassarov commented Oct 6, 2022

fmuyassarov commented Oct 6, 2022

fmuyassarov commented Oct 8, 2022

fmuyassarov commented Oct 8, 2022

fmuyassarov commented Oct 9, 2022

fmuyassarov commented Oct 10, 2022

marquiz left a comment

marquiz commented Oct 13, 2022

fmuyassarov commented Oct 13, 2022

marquiz commented Dec 2, 2022

fmuyassarov commented Dec 2, 2022

marquiz left a comment

ArangoGutierrez left a comment

k8s-ci-robot commented Dec 6, 2022

Allow optionally setting node taints defined on the NodeFeatureRule CR #910

Allow optionally setting node taints defined on the NodeFeatureRule CR #910

Conversation

fmuyassarov commented Oct 6, 2022 • edited Loading

netlify bot commented Oct 6, 2022 • edited Loading

✅ Deploy Preview for kubernetes-sigs-nfd ready!

fmuyassarov commented Oct 6, 2022

fmuyassarov commented Oct 6, 2022

fmuyassarov commented Oct 8, 2022

fmuyassarov commented Oct 8, 2022

fmuyassarov commented Oct 9, 2022

fmuyassarov commented Oct 10, 2022

marquiz left a comment

Choose a reason for hiding this comment

marquiz commented Oct 13, 2022

fmuyassarov commented Oct 13, 2022

marquiz commented Dec 2, 2022

fmuyassarov commented Dec 2, 2022

marquiz left a comment

Choose a reason for hiding this comment

ArangoGutierrez left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Dec 6, 2022

fmuyassarov commented Oct 6, 2022 •

edited

Loading

netlify bot commented Oct 6, 2022 •

edited

Loading