v1.9.0 Release

v1.9.0

The CNI v1.9.0 release will support higher pod density per node and also reduces the number of EC2 calls to create and attach more ENIs by leveraging the recent EC2 feature - Assigning prefixes to Amazon EC2 network interfaces. This feature is only supported on "nitro" based instances. Each secondary IP will be replaced by a (/28) prefix and IPAMD will derive a (/32) IP from these prefixes for pod IP allocation. Number of prefixes which can be attached to the ENI is still limited by the IP addresses per network interface per instance type- for instance a t3.medium can have 5 (/28)prefixes per ENI. Since the number of pods per node will increase, make sure the max pods are adjusted appropriately and keep in mind CPU and memory constraints will limit the number of pods that be deployed on a node. To help with the max pods computation we have added this helper script.

To enable the feature, set ENABLE_PREFIX_DELEGATION to true. This environment variable can be set to true or false while pods are running but kubelet maxPods has to be manually changed.

PS: This feature is available in all public regions and AWS GovCloud (US), with support in China (Beijing), and China (Ningxia) coming soon.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

With this feature we support WARM_PREFIX_TARGET and IPAMD will keep a free prefix in warm pool. We also support WARM_IP_TARGET and MINIMUM_IP_TARGET and IPAMD will allocate a new prefix if the existing prefixes are not sufficient to maintain the warm pool. This document talks about how these 3 warm targets impact the number of IPs available in the warm pool.

Since the IPs with this feature will be allocated from the prefixes, we won't be able to support downgrades. Also WARM_ENI_TARGET won't be supported with ENABLE_PREFIX_DELEGATION set to true.

Changes since v1.8.0:

• Enhancement - EC2 sdk model override (#1508, @jayanthvn)
• Enhancement - Prefix Delegation feature support (#1516, @jayanthvn)
• Enhancement - Header formatting for env variable (#1522, @jayanthvn)
• Enhancement - non-nitro instances init issues (#1527, @jayanthvn)
• Enhancement - Add metrics for total prefix count and ips used per cidr (#1530, @jayanthvn)
• Enhancement - Update documentation for PD (#1540, @jayanthvn)
• Enhancement - Update SDK Go version (#1544, @jayanthvn)

To apply this release:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.9.0/config/v1.9/aws-k8s-cni.yaml

Verify the update:

$ kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2                                                   
amazon-k8s-cni-init:v1.9.0
amazon-k8s-cni:v1.9.0

v1.8.0 Release

v1.8.0

Changes since v1.7.10:

• Bug - Use symmetric return path for non-VPC traffic - alternate solution (#1475, @kishorj)
• Bug - Gracefully handle failed ENI SG update (#1341, @jayanthvn)
• Bug - Fix CNI crashing when there is no available IP addresses (#1499, @M00nF1sh)
• Bug - Use primary ENI SGs if SG is null for Custom networking (#1259, @jayanthvn)
• Bug - Don't cache dynamic VPC IPv4 CIDR info (#1113, @anguslees)
• Improvement - Address Excessive API Server calls from CNI Pods (#1419, @achevuru)
• Improvement - refine ENI tagging logic (#1482, @M00nF1sh)
• Improvement - Change tryAssignIPs to assign up to configured WARM_IP_TARGET (#1279, @jacksontj)
• Improvement - Use regional STS endpoint (#1332, @nithu0115)
• Improvement - Update containernetworking dependencies (#1200, @mogren)
• Improvement - Split Calico manifest into two (#1410, @caseydavenport)
• Improvement - Update Calico manifest to support ARM & AMD (#1282, @jayanthvn)
• Improvement - Auto gen of AWS CNI, metrics helper and calico artifacts through helm (#1271, @jayanthvn)
• Improvement - Refactor EC2 Metadata IMDS code (#1225, @anguslees)
• Improvement - Unnecessary logging for each CNI invocation (#1469, @jayanthvn)
• Improvement - New instance types (#1463, @jayanthvn)
• Improvement - Use 'exec' ENTRYPOINTs (#1432, @anguslees)
• Improvement - Fix logging texts for ENI cleanup (#1209, @mogren)
• Improvement - Remove Duplicated vlan IPTable rules (#1208, @mogren)
• Improvement - Minor code cleanup (#1198, @mogren)
• HelmChart - Adding flags to support overriding container runtime endpoint. (#1443, @haouc)
• HelmChart - Add podLabels to amazon-vpc-cni chart (#1440, @haouc)
• HelmChart - Add workflow to sync aws-vpc-cni helm chart to eks-charts (#1430, @fawadkhaliq)
• Testing - Remove validation of VPC CIDRs from ip rules (#1476, @kishorj)
• Testing - Updated agent version (#1474, @cgchinmay)
• Testing - Fix for CI failure (#1470, @achevuru)
• Testing - Binary for mtu and veth prefix check (#1458, @cgchinmay)
• Testing - add test to verify cni-metrics-helper puts metrics to CW (#1461, @abhipth)
• Testing - add e2e test for security group for pods (#1459, @abhipth)
• Testing - Added Test cases for EnvVars check on CNI daemonset (#1431, @cgchinmay)
• Testing - add test to verify host networking setup & cleanup (#1457, @abhipth)
• Testing - Runners failing because of docker permissions (#1456, @jayanthvn)
• Testing - decouple test helper input struct from netlink library (#1455, @abhipth)
• Testing - add custom networking e2e test suite (#1445, @abhipth)
• Testing - add integration test for ipamd env variables (#1453, @abhipth)
• Testing - add agent for testing pod networking (#1448, @abhipth)
• Testing - fix format of commited code to fix unit test step (#1449, @abhipth)
• Testing - Unblocks Github Action Integration Tests (#1435, @couralex6)
• Testing - add warm ENI/IP target integration tests (#1438, @abhipth)
• Testing - add service connectivity test (#1436, @abhipth)
• Testing - add network connectivity test (#1424, @abhipth)
• Testing - add ginkgo automation framework (#1416, @abhipth)
• Testing - Add some test coverage to allocating ENIs (#1234, @mogren)
• Testing - Add some minimal tests to metrics (#1228, @mogren)

Thanks to all the contributors ❤️ !!!

To apply this release:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.8.0/config/v1.8/aws-k8s-cni.yaml

Verify the update:

$ kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2                                                   
amazon-k8s-cni-init:v1.8.0
amazon-k8s-cni:v1.8.0

v1.7.10 Release

v1.7.10

Changes since v1.7.9:

• Improvement - Multi card support - Prevent route override for primary ENI across multi-cards ENAs (#1396 by jayanthvn)

Currently P4 family supports more than one network card and IPAMD will ignore ENIs on non-zero network cards. IPAMD will continue to manage ENI allocation on network card 0 but will not manage ENIs on non-zero network cards.

To apply this release:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.10/config/v1.7/aws-k8s-cni.yaml

Verify the update:

$ kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2                                                   
amazon-k8s-cni-init:v1.7.10
amazon-k8s-cni:v1.7.10

1.7.10 Release Candidate 1

v1.7.10-rc1

Changes since v1.7.9:

• Improvement - Multi card support - Prevent route override for primary ENI across multi-cards ENAs (#1396 by jayanthvn)

Currently P4 family supports more than one network card and IPAMD will ignore ENIs on non-zero network cards. IPAMD will continue to manage ENI allocation on network card 0 but will not manage ENIs on non-zero network cards.

To apply this release:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.10-rc1/config/v1.7/aws-k8s-cni.yaml

Verify the update:

$ kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2                                                   
amazon-k8s-cni-init:v1.7.10-rc1
amazon-k8s-cni:v1.7.10-rc1

v1.7.9 Release

v1.7.9

Changes since v1.7.8:

• Improvement - Adds http timeout to aws sessions (#1370 by couralex6)
• Improvement - Switch calico to be deployed with the Tigera operator (#1297 by tmjd)
• Improvement - Update calico to v3.17.1 (#1328 by lwr20)
• Improvement - update plugins to v0.9.0 (#1362 by fr0stbyte)
• Improvement - update github.com/containernetworking/plugins to v0.9.0 (#1350 by fr0stbyte)
• Bug - Fix regex match for getting primary interface (#1311 by Jayanthvn)
• Bug - Output to stderr when no log file path is passed (#1275 by couralex6)
• Bug - Fix deletion of hostVeth rule for pods using security group (#1376 by SaranBalaji90)

To apply this release:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.9/config/v1.7/aws-k8s-cni.yaml

Verify the update:

$ kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2                                                   
amazon-k8s-cni-init:v1.7.9
amazon-k8s-cni:v1.7.9

v1.7.9 Release Candidate 1

v1.7.9 Release Candidate 1

Changes since v1.7.8:

• Improvement - Adds http timeout to aws sessions (#1370 by couralex6)
• Improvement - Switch calico to be deployed with the Tigera operator (#1297 by tmjd)
• Improvement - Update calico to v3.17.1 (#1328 by lwr20)
• Improvement - update plugins to v0.9.0 (#1362 by fr0stbyte)
• Improvement - update github.com/containernetworking/plugins to v0.9.0 (#1350 by fr0stbyte)
• Bug - Fix regex match for getting primary interface (#1311 by Jayanthvn)
• Bug - Output to stderr when no log file path is passed (#1275 by couralex6)
• Bug - Fix deletion of hostVeth rule for pods using security group (#1376 by SaranBalaji90)

To apply this release:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.7/config/v1.7/release-candidate/aws-k8s-cni.yaml

Verify the update:

$ kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2                                                   
amazon-k8s-cni-init:v1.7.9-rc1
amazon-k8s-cni:v1.7.9-rc1

v1.7.8 Release

This is a patch release to make EC2 DescribeNetworkInterfaces calls paginated and avoid EC2 API call latency in some cases.

Changes since v1.7.7 -

• Improvement - Replace DescribeNetworkInterfaces with paginated version (#1333, @haouc)

If you want to apply this config to one of your clusters:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.8/config/v1.7/aws-k8s-cni.yaml

Verify the update:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.8
amazon-k8s-cni:v1.7.8

To use version v1.7.8 of the cni-metrics-helper in a cluster:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.8/config/v1.7/cni-metrics-helper.yaml

v1.7.7 Release

This is a patch release to cleanly delete regular pods with PPSG enabled with force delete.

Changes since v1.7.6 -

• Bug - Rearrange Pod deletion workflow (#1315, @SaranBalaji90)

Thanks to @SaranBalaji90 for the fix.

If you want to apply this config to one of your clusters:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.7/config/v1.7/aws-k8s-cni.yaml

Verify the update:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.7
amazon-k8s-cni:v1.7.7

To use version v1.7.7 of the cni-metrics-helper in a cluster:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.7/config/v1.7/cni-metrics-helper.yaml

v1.7.6 Release

This is a patch release to support instances with EFA ENIs (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html).

Changes since v1.7.5

• Improvement - Avoid detaching EFA ENIs (#1237 , @mogren)
• Improvement - Add t4g instance type (#1219 , @mogren)
• Improvement - Add p4d.24xlarge instance type (#1238 , @mogren)
• Improvement - Update calico to v3.16.2 (#1235 , @lwr20)
• Improvement - Update readme on stdout support for plugin log file (#1251 , @jayanthvn)
• Bug - Make p3dn.24xlarge examples more realistic (#1263 , @mogren)
• Bug - Make sure we have space for a trunk ENI (#1210 , @mogren)
• Bug - Update README for DISABLE_TCP_EARLY_DEMUX (#1273 , @SaranBalaji90)
• Bug - Update p4 instance limits (#1289 , @jayanthvn)

Thanks to all the contributors!!!

If you want to apply this config to one of your clusters:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.6/config/v1.7/aws-k8s-cni.yaml

Verify the update:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2

amazon-k8s-cni-init:v1.7.6
amazon-k8s-cni:v1.7.6

To use version v1.7.6 of the cni-metrics-helper in a cluster:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.6/config/v1.7/cni-metrics-helper.yaml

v1.7.5 Release

This is a patch release to address issue #1246, #1250.

The issue is caused by init.sh script tries to grep host IP and could match more than one ENIs. The consequence is the CNI starting up will be failed.

Changes since v1.7.4

• Bug - Match primary ENI IP correctly (#1247 , @mogren)

If you want to apply this config to one of your clusters:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.5/config/v1.7/aws-k8s-cni.yaml

Verify the update:

kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2

amazon-k8s-cni-init:v1.7.5
amazon-k8s-cni:v1.7.5

To use version v1.7.5 of the cni-metrics-helper in a cluster:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.5/config/v1.7/cni-metrics-helper.yaml