amazon-ec2-net-utils incompatible with amazon-vpc-cni-k8s

[dims]

Following up on problem noted here:
awslabs/amazon-eks-ami#1528 (comment)

Context:
amazon-vpc-cni-k8s is a networking plugin for Kubernetes, it is used by a lot of installers for kubernetes on EC2 instances including community based tools like kops etc. amazon-vpc-cni-k8s is developed and maintained by the Amazon EKS team members and primary use case is definitely for EKS. The link above points to the issue and discussion on when we identified the problem and removed the package. This happened on AL2 a while ago and cropped up yet again in AL2023 because the package name changed just a bit (amazon- prefix was added). So we had to hunt again for the root cause and ended up tagging the same package maintained in this repo as the cause. An overview of how the amazon-vpc-cni-k8s works is here. essentially the ip address routing rules setup by kubernetes/containerd/amazon-vpc-cni-k8s get stomped on by amazon-ec2-net-utils.

[dims]

cc @jayanthvn @orsenthil @bryantbiggs @achevuru

[nmeyerhans]

I don't see a corresponding issue within the amazon-vpc-cni-k8s repository, but I think there should be one as that's where the integration will need to happen. Net-utils shouldn't need to be aware of optional additional components that may be layered on top of it.

net-utils utilizes udev triggers to generate configuration for systemd and systemd-networkd. Each of these components provides facilities to override existing configuration.

A couple of options come to mind in terms of approaches here:

1. If amazon-vpc-cni-k8s wants to deploy a big hammer and disable everything that net-utils does, it could simply install an empty /etc/udev/rules.d/99-vpc-policy-routes.rules. This suppresses everything that net-utils does, and in terms of interface configuration is functionally equivalent of removing it.
2. Alternatively, it could deploy more targeted logic and install custom networkd configuration, e.g.

[ec2-user@ip-10-0-0-46 ~]$ cat /etc/udev/rules.d/98-vpc-override.rules 
SUBSYSTEM=="net", ACTION=="add", ENV{ID_NET_DRIVER}=="vif|ena|ixgbevf", RUN="/usr/local/bin/eni.sh"
[ec2-user@ip-10-0-0-46 ~]$ cat /usr/local/bin/eni.sh 
#!/bin/sh
# suppress configuration of hotplugged ENI interfaces

env | logger -t net-info -p user.debug

mkdir -p /run/systemd/network

cat <<EOF > "/run/systemd/network/20_${INTERFACE}.network"
[Match]
Driver=ena ixgbevf vif
Name=${INTERFACE}

[Link]
MTUBytes=9001

[Network]
DHCP=no
EOF

networkctl reload

[stewartsmith]

As @nmeyerhans mentions, It looks like amazon-vpc-cni-k8s is bypassing all the systemd-networkd configuration and doing a lot of things itself.

It feels like it should move to poking the right things into the systemd-networkd config and having it remain compatible with the host systemd-networkd configuration. This likely would fix issues related to other systemd-networkd base OSs as well (it looks like there's some other issues that have been filed against the amazon-vpc-cni-k8s package in the past few years on this).

It looks like the configuration comes down to a few things, and systemd-networkd certainly seems to support everything it's doing. A quick glance through the code seems to match up everything that systemd-networkd.service points to: primarily systemd.network, but also systemd.link and systemd.netdev .