CoreDNS fails valid lookups as soon as an invalid lookup is made

[DimitrijeManic]

CoreDNS seems to break for a few seconds every time there is an invalid nslookup.

minikube version: v1.20.0
commit: c61663e

coredns version: k8s.gcr.io/coredns:1.7.0 (I have tried k8s.gcr.io/coredns/coredns:v1.8.0 and had the same issues)

k8s version: v1.20.0

Steps to reproduce the issue:

1. minikube start
2. show core-dns logs

klo -l k8s-app=kube-dns -n kube-system

3. In a separate terminal run dnstools & request a valid lookup

kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools-valid

watch -n 2 nslookup google.com

4. In a separate terminal run dnstools & requst an invalid lookup

kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools-invalid

nslookup asdf.asdf

Full output of minikube logs command:

[ERROR] plugin/errors: 2 asdf.asdf. A: read udp 172.17.0.8:37408->192.168.65.1:53: i/o timeout
[ERROR] plugin/errors: 2 asdf.asdf. A: read udp 172.17.0.8:49789->192.168.65.1:53: i/o timeout
[ERROR] plugin/errors: 2 google.com. A: read udp 172.17.0.8:33396->192.168.65.1:53: i/o timeout
[ERROR] plugin/errors: 2 google.com. A: read udp 172.17.0.8:57661->192.168.65.1:53: i/o timeout
[ERROR] plugin/errors: 2 google.com. A: read udp 172.17.0.8:57512->192.168.65.1:53: i/o timeout
[ERROR] plugin/errors: 2 google.com. A: read udp 172.17.0.8:42618->192.168.65.1:53: i/o timeout
[ERROR] plugin/errors: 2 google.com. A: read udp 172.17.0.8:53487->192.168.65.1:53: i/o timeout

Full output of failed command:

[DimitrijeManic]

I believe this is the issue and is discussed here weaveworks/weave#3287

[RA489]

/kind support

[medyagh]

@DimitrijeManic I am not fimmiliar with this issue
@sadlil might have better insight on this ?

I am curious did u get tot the root cause or find out more about it ?

[DimitrijeManic]

@medyagh The problem turned out to be a service we were using (Cisco Umbrella) which would take over port 53 and cause issues with ingress & ingress-dns.
Related to issue #3036