-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Long session connections get dropped #144
Comments
Can you please set this flag -> |
I enabled this flag and now I am getting logs in the nodeagent container but all the logs looks like:
There is no DENY verdict in those logs, I don't know why the nodeagent print the logs like this. any ideas why? |
Yes it looks similar. v1.0.7-rc1 tag is available. You can replace the aws-eks-nodeagent container image on aws-node DS with the v1.0.7-rc1 tag For example -
Please try and let us know if it is holding up. |
Still not working for me, mongodb disconnecting every 5 minutes. |
I missed the above logs and those are just the pod logs. Can you share the node logs so you can run this script |
After furthere investigation, it seems to be that the long session connections got terminated because of istio envoy sidecar. Does this new information help? How to reproduce: |
Will you be able to try this image -
Please make sure you replace the account number and region. |
Discussed with @Rez0k offline and the official release, i.e. VPC CNI v1.15.5 containing Network Policy agent tag v1.0.7, should fix this issue. Waiting for confirmation before closing issue |
I will try it this week and will update here. |
update: I prefer to wait for your next release candidate according to: #175 (comment) |
@Rez0k - We have |
@Rez0k - Did you get a chance to verify the image? |
I prefer to wait for the official release, I will try on the v1.0.8 release. |
v1.0.8 release is available - https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.16.3. Please try it out and let us know if you see any issues.. |
Seems to work! |
What happened:
After migrating from Calico to aws vpc cni network policies (we are working with istio if that matters) we experience disconnections on long sessions connections such as Redis pub-sub or MongoDB connections.
The connection gets closed and then it reconnects again, which happens every few minutes.
I configured the vpc cni addon to be:
I can't see any logs in the nodeagent container of the aws-node pod, all I get is:
So, I can't attach logs.
What you expected to happen:
I expect the connections not to be dropped from the first place
How to reproduce it (as minimally and precisely as possible):
Install VPC CNI (network policy enabled) on eks 1.28, apply network policy and try to connect to mongodb instance (or redis pub sub) or probably any other long session technology.
The image I am using is: public.ecr.aws/docker/library/node:18.16.0-bullseye-slim
Run this sample code on a nodejs pod (prefarable to be public.ecr.aws/docker/library/node:18.16.0-bullseye-slim):
Wait few minutes and you should see logs like:
Anything else we need to know?:
I use istio in my cluster and used Calico up until yesterday, I terminated all instances to flush all leftovers from Calico.
With Calico everything worked as expected.
Environment:
kubectl version
): v1.28.3-eks-4f4795dcat /etc/os-release
): Debian GNU/Linux 11 (bullseye)uname -a
): Linux #### ####.amzn2.x86_64The text was updated successfully, but these errors were encountered: