input/output error running on Celeron k3s node

[adampetrovic]

Recently upgraded to NFD from v0.14.1 -> v0.14.3

All nodes are running v1.28.2+k3s1

I have noticed that it's running fine on all my other k3s nodes which are i5 / i7-based NUCs however I get constant CrashLoopBackOffs on the single node that is a Intel(R) Celeron(R) J4005 CPU.

node-feature-discovery-gc-774b55db5b-hwvw9          1/1     Running            0               11d     10.42.6.20    k8s-node-7   <none>           <none>
node-feature-discovery-master-6f7c7ddd4f-m9xpq      1/1     Running            0               11d     10.42.6.97    k8s-node-7   <none>           <none>
node-feature-discovery-worker-2pwgw                 1/1     Running            2 (3m39s ago)   11d     10.42.2.184   k8s-node-3   <none>           <none>
node-feature-discovery-worker-6vz48                 1/1     Running            1 (3d5h ago)    11d     10.42.1.226   k8s-node-1   <none>           <none>
node-feature-discovery-worker-cm2w9                 1/1     Running            0               11d     10.42.0.76    k8s-node-2   <none>           <none>
node-feature-discovery-worker-mkll6                 1/1     Running            0               11d     10.42.5.8     k8s-node-6   <none>           <none>
node-feature-discovery-worker-nt6fx                 0/1     CrashLoopBackOff   3 (22s ago)     63s     10.42.4.159   k8s-node-5   <none>           <none>
node-feature-discovery-worker-vwjx8                 1/1     Running            0               11d     10.42.3.238   k8s-node-4   <none>           <none>
node-feature-discovery-worker-znf9z                 1/1     Running            1 (5d ago)      11d     10.42.6.140   k8s-node-7   <none>           <none>

➜ k logs pod/node-feature-discovery-worker-nt6fx -n kube-system
exec /usr/bin/nfd-worker: input/output error

[marquiz]

@adampetrovic, looks really strange 🧐 Is the node healthy are you low on disk space or ram? Does the kernel dmesg give any insight?

[adampetrovic]

All looking normal.

31G free on the root partition, 2.8GB free memory

Reverting back to v0.14.1 fixes it, so it seems to be change related.

[marquiz]

I can't think of anything else that there's a problem in that specific node. Isn't there anything in dmesg or system log/journal about the error? Have you tried other versions of NFD like v0.14.2 or v0.13.6?

[marquiz]

@adampetrovic are you still experiencing this problem? Have you tried with latest v0.14 (or v0.15)?

[adampetrovic]

Thanks for reminding me! It turns out the problem was a container image download that got corrupted, leaving the binaries in the image empty. Deleting and redownloading solved the problem. Feel free to close this off