DNS intermittent delays of 1/2s with nodelocaldns

[123BLiN]

We have an issue that we are troubleshooting almost a week for now related to the well known issue with DNS conntrack race conditions ( kubernetes/kubernetes#56903 ) and I hope someone hits it too and can at least confirm we are not alone here.

• We have applied node-local-dns setup that was intended to mitigate all races in conntrack due to the fact that it listens on the local node IP and traffic never hits conntrack module however we still see the same timeout related to DNS searches.
• fresh KOPS 1.16.3 setup in AWS with default debian stretch worker nodes (without fixes to iptables)
• latest (1.15.12) node local dns in IPTABLES mode according to https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/
• all iptables rules related to local DNS traffic are present with NOTRACK and the more they have hits in iptables -L
• conntrack -S shows almost none insert_failed on node (at least not that much as we see delays and numbers are pretty static)
  - conntrack -L shows no connections with src/dst IP/port of DNS udp traffic, only tcp connections to the upstream CoreDNS - so I assume it all goes directly to the node cache
• traffic to upstream CoreDNS is upgraded to TCP according to CoreDNS logs - no UDP here also timings are all good
• same in node local CoreDNS logs - all responses have ms delays

however we still see in jaeger intermitent 2s delays in some calls (PHP app, new process and new DNS search per request), when I set DNSOptions in pod to single-request-reopen, ndots 2 and timeout 1 delays are still present but now they are all 1s so this all looks like some issue with DNS traffic from pod to node-cache but I can't figure out what else to look here.

[prameshj]

Any logs in node-local-dns pods? There will be error messages if node-local-dns does not get a response back from coreDNS upstream within 2s.

[123BLiN]

No errors in node-local-dns logs - I've set them to log everything.
Also today I've tried next steps without success:

• Change DNS config in pods to use-vs - it works as expected - all requests in node-local-dns are now TCP but delays are still present from time to time and are still 2s
• Change DNS opts timeout to 5s - delays are present and are 2s %)

This forwards me to the though, that maybe it is not DNS related at all (basically they are all redis calls from predis PHP library)
But in Jaeger I can only see the whole request that includes DNS lookup + redis call itself, so maybe I should forward requests to some TCP proxy and see timings there or perform some tcpdump-fu :(

I'm going to take another round of testing tomorrow to confirm DNS related problem and will close this issue if not.
Sorry for disturbing, and thanks for your reply!

[inter169]

maybe the app (jaeger) was using some invalid FQDN in the DNS query auto-completed by underlying C runtime library (libc), or due to some incorrect DNS settings,
doing tcpdump or strace against the app is helpful.

[123BLiN]

I'm going to dig into tcpdump, but this will take time, however I was able to make a simple test - change DNS domain name of the redis service to the concrete IP address in service configs - this removed any delays completely.
The name is external (redis.example.com) and is hosted in AWS Route53 zone.

[123BLiN]

We have found the cause of delays for our setup and it is DNS (as always) 😄
I've made a couple of tcpdumps - one from the service pod and one from node-local-dns on the same node, in the first one it is clear that some responses from node-local-dns are slow and are 1,2 and 4 seconds. In the second dump I've probably found a reason for this behaviour - at the same microtime of slow 10-20 responses to the service pod I see slow responses from internal AWS Route53 resolver (10.2.0.2 in our case).
What is interesting here is that it seems like one delayed response from AWS delays multiple simultaneous parallel requests from the service, in our case PHP service does not have its own cache so every request from outside produces a lot of internal requests to the DB, redis, elasic etc. and every time it tries to resolve all of them, and the more I see a lot of NXDOMAIN responses from the AWS for the search domains that adds local node resolver, and maybe we are hitting some AWS limits because of that.

I'm closing the issue as node-local-dns setup works as expected here, however I have a few questions and I'll be grateful if someone can clarify them:

• Default node-local-dns setup replaces __PILLAR__UPSTREAM__SERVERS__ with /etc/resolv.conf in our case, probably I should replace it with 10.2.0.2 manually to remove search domains issue completely? Also should I remove force_tcp from forwarder config?
• Is there a way to tell coredns to resolve domains from cache in advance in order to always have a fresh cached records locally? 🤪
• It is interesting what is the logic for incoming parallel requests to resolve the same DNS name if cache is missed - are they all waiting for one single resolve in upstream or coredns makes separate requests?

[prameshj]

Default node-local-dns setup replaces PILLAR__UPSTREAM__SERVERS with /etc/resolv.conf in our case, probably I should replace it with 10.2.0.2 manually to remove search domains issue completely? Also should I remove force_tcp from forwarder config?

I am not sure this will fix the issue. How do requests from pods reach AWS resolver(10.2.02) today? Has it been configured as a stubDomain? Replacing the upstream server will not disable searchpath expansion, not sure if that's the intent.
You can remove force_tcp from the config for everything except "cluster.local" block. We did make this change in the recent node-cache image.

Is there a way to tell coredns to resolve domains from cache in advance in order to always have a fresh cached records locally? 🤪

You can use the prefetch option for this - https://coredns.io/plugins/cache/

It is interesting what is the logic for incoming parallel requests to resolve the same DNS name if cache is missed - are they all waiting for one single resolve in upstream or coredns makes separate requests?

I believe it would open separate requests for it, since upon cache miss, control goes to the next plugin, which I think will be "forward" in this case. https://github.com/coredns/coredns/blob/614d08cba29ed4904d11008e795c081c4f392b77/plugin/cache/handler.go#L35
@rajansandeep can you confirm? Thanks!

[chrisohaver]

I believe it would open separate requests for it ...

That's correct.

[123BLiN]

Thank you very much for your answers and especially for prefetch option this is awesome feature!
I think we have found a way to go for us - it is another CoreDNS plugin https://coredns.io/plugins/route53/
as we have all popular records in the same internal zone, but we will check prefetch if route53 plugin won't work for us.