using amazon-vpc-cni-k8s outside eks
#2839
Comments
|
We are running the AWS CNI outside of EKS. We also have the AWS credential provider installed, this allows the kubelet to use the instance credentials to pull from private ECR registries. Before Kubernetes 1.28 (I think, might be off by a version), this functionality was bundled as part of the kubelet. |
|
That's intresting @kwohlfahrt! I've never used
|
AFAIK, the credential provider can't be installed by applying manifests, it must be installed to your node, since you must change the kubelet flags to use it. The binary and configuration must be placed on disk, and then the kubelet's flags have to be modified to point to the configuration, and the path to search for the binary. This is documented on this page, which also includes an example config.
Pre-built binaries can be found here (source)
Yes, we've used it with containerd in the past, though we are using cri-o now. AFAIK, the container runtime never interacts with the credential provider directly - the credential provider is called by the kubelet, which then passes the received credentials on to your container runtime. So it shouldn't matter whether you are using containerd, crio, etc. |
|
Thank you so much @kwohlfahrt! I was able to follow through and resolve this and all the pods are successfully running now. These are the steps I took,
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- name: ecr-credential-provider
matchImages:
- "*.dkr.ecr.*.amazonaws.com"
defaultCacheDuration: "12h"
apiVersion: credentialprovider.kubelet.k8s.io/v1
env:
|
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
|
Hey @kwohlfahrt! It seems this wasn't resolved entirely. As soon as I joined another node I ran into troubles with
I did assign |
|
Hm, I'm not sure. My only suspicion is you might be hitting #2840 I reported the other day. You can easily check by connecting to your node and seeing if You can see the linked PR in that issue for the fix (the volume needs to be defined with |
|
Thank You @kwohlfahrt! I had some missing [ERROR] plugin/errors: 2 4999722014791650549.7690820414208347954. HINFO: read udp 10.0.43.148:57589->10.0.0.2:53: i/o timeout
[ERROR] plugin/errors: 2 4999722014791650549.7690820414208347954. HINFO: read udp 10.0.43.148:38940->10.0.0.2:53: i/o timeout
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.27.4/tools/cache/reflector.go:231: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout |
|
Update! I was really unable to resolve
I switched to cilium and let go of my dream to connect k8s and aws. |
This seems like the coredns pod go the ip-ddress, but it wasn't able to communicate with the API server, due to missing permissions? The nodes/pods should have the ability to communicate with API server with the necessary permissions. Were you able to narrow down to any permission issue? |
Not really! I really did all I could and scanned all of journalctl to find something. I wrote about it here & I couldn't get
|
|
I am hitting the same issue. the Pod cannot communicate with any endpoints including
|
|
@terryjix - This is question on setting up VPC CNI on a non EKS cluster. How did you go about with this? |
|
Closing this due to lack of more information. |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
|
This issue needs to be reopened - it seems to be a fairly ubiquitous issue when attempting to use the amazon-vpc-cni in a non-EKS environment. I've also encountered it (coredns not able to communicate): [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server |
@orsenthil Why was this closed? It seems like there's plenty of information and repro steps? |
We will need to reproduce this and investigate. Re-opened. |
|
Thanks! I've got a cluster that reproduces and willing to screen share/support as needed. |
|
I've fixed my issue by running vpc-cni-k8s on EKS optimized AMI. vpc-cni-k8s plugin conflicts with ec2-net-utils. ec2-net-utils adds more route rules which broke the pod to pod communication in my case. the EKS optimized ami has optimized this issue. |
Does this work for even outside EKS? I think this bug was for outside EKS (for example, I'm running self-managed on ubuntu AMIs with kubeadm) |
|
yes, I used kubeadmin to create kubernetes cluster on Amazon Linux 2 ami and found the pod cannot communicate with outside. some strange rules created on route table which overwrites the rules vpc-cni created. You can find optimized ubuntu ami from https://cloud-images.ubuntu.com/aws-eks/ . Maybe it can fix your issue. You can build your self-managed kubernetes control plan on these amis. The optimized ami has disabled some services may affect network configuration in the OS. |
It says clearly on the page: These images are customised specifically for the EKS service, and are not intended as general OS images. |
What happened:
Hi! I have an
ec2instance &containerdas the container runtime inside a private subnet (which has outbound internet access) inap-south-1. I have intialized a new cluster withkubeadm initon this master node. It ran successfully. I then wanted to installamazon-vpc-cnias the network manager for my k8s cluster. I rankubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/master/aws-k8s-cni.yamland checked the pods inkubectl get pods -n kube-system. One of the pod created byamazon-vpc-cni-k8snamedaws-node-xxxxthrows an error when trying to initialise. I didkubectl describe pod aws-node-xxx -n kube-systemand I get the following.I don't understand why this fails. Is it not possible to use
amazon-vpc-cnioutside eks in self managed cluster? I also looked around here in issues & it seems like other people had this issue before but I was unable to resolve it myself. Here is my policyk8s_master_ecrinside ak8s_masterrole which is connected to thismasterinstance via an instance profile,{ "Version": "2012-10-17", "Statement": [ { "Sid": "K8sECR", "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:GetRepositoryPolicy", "ecr:DescribeRepositories", "ecr:ListImages", "ecr:BatchGetImage" ], "Resource": "*" } ] }Environment:
kubectl version):masterbranchcat /etc/os-release):uname -a):Linux ip-x-x-x-x.ap-south-1.compute.internal 6.1.0-13-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/LinuxThe text was updated successfully, but these errors were encountered: