-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable fully random masquerading ports #1001
Enable fully random masquerading ports #1001
Conversation
There is a race condition in linux which can lead to packages being dropped when many connections are established to the same address. See this for more datail: https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02 This commit introduces the proposed workaround of enabling fully randomized ports. Since this requires iptables 1.6.2, it requires using alpine:edge until the next release.
Honestly, I'm not sure this helps. I've just tested this on a cluster here and I don't see the conntrack insert failures going down at all.. |
Hi @discordianfish |
@maxlaverse Oh Hi! :) No, conntrack table isn't full. There are at least ~100k entries available on all my nodes:
|
As discussed together, I don't know what the issue could be here nor why the Our plan is still to get rid of NAT completely |
I read a few issues about those DNS timeouts. As pointed out in weaveworks/weave#3287 (comment), the problem in that case might be with the DNAT rules that transform the Service IP into a PodIP in the requests for nameserver resolution. It looks like there is the same delay as with SNAT, between the allocation of a conntrack record and its insertion in the table. This can lead some collisation and ultimately in packet drops. Apparently, IPVS uses its own connection tracking module (http://kb.linuxvirtualserver.org/wiki/Performance_and_Tuning). If you have a network that supports routable PodIPs and switch kube-proxy to ipvs, it might help with your issue (if it's really about DNAT in conntrack). Just read there are different issues with ipvs too: kubernetes/kubernetes#57841 |
@discordianfish do you deployed a successful fix? @maxlaverse we also have problems with that and run coreos + flannel in 85 clusters. Monitoring shows regular spikes of DNS request timeouts.
We also increased node_nf_conntrack_entries_limit and alert on node_nf_conntrack_entries, we do not see any issues there. CoreOS release
|
Hi @szuecs @Quentin-M wrote an article about it that might interest you, with a workaround: https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/ I hope it will help |
@maxlaverse thanks for the response. I think also that DNAT will be the problem. I ported basically the code by @Quentin-M to flannel (we use 53 as destination so also change 5353 to 53, but anything else is pretty similar). I added this as sidecar to flannel running as daemonset and added:
I published the code repository at https://github.com/szuecs/flannel-tc I started the test at 6PM on 7/9 and it looks better than before (so the race condition is less likely to happen than before). We have to wait before to claim a success: |
…hat the kernel race condition happens less likely. Background see https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/ kubernetes/kubernetes#62628 flannel-io/flannel#1001
Thanks @discordianfish for the PR and @maxlaverse and @szuecs for the additional diags. Now that iptables v1.6.2 is in the latest Alpine release I'm going to take the fix from #1040, so closing this one. |
Description
There is a race condition in linux which can lead to packages being
dropped when many connections are established to the same address.
See this for more datail: https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02
This commit introduces the proposed workaround of enabling fully
randomized ports. Since this requires iptables 1.6.2, it requires using
alpine:edge until the next release.
See this for similar discussion in weave: weaveworks/weave#3287