kubeadm: Control Plane with "ClusterFirstWithHostNet" is a circular dependency.

[mauilion]

BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.13

What happened?

I was looking through the changelog for 1.13 and noticed:
kubernetes/kubernetes#68890

With this change the apiserver will not be able to resolv dns names until kube-proxy and kube-dns are started
.
The resolv.conf of the apiserver when ClusterFirstWithHostNet is set:

bash-4.3# cat /etc/resolv.conf 
nameserver 10.96.0.10 #service ip of kube-dns
search default.svc.cluster.local svc.cluster.local cluster.local cisco.com
options ndots:5

This is likely green because the apiserver manifest has the following set:

--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
--etcd-servers=https://127.0.0.1:2379

With these settings we would probably make the tests green because kube-apiserver will be able to resolve what it needs to by using the ip addresses bypassing dns resolution.

If the user has an external etcd cluster this will break as the apiserver will be unable to resolve the external etcd cluster by name.

If the kubelet-preferred-address-types is the default (Hostname) this will break as well.

Any other calls the apiserver has to make before the kube-dns and kube-proxy are started will fail (Though there probably aren't too many calls to make there.)

[mauilion]

/assign @chuckha
/priority critical
@timothysc

[rdodev]

Hmmmm 👁

[chuckha]

I reproduced both the failure using the release-1.13 release branch and the the successful case using release-1.12.

[chuckha]

Method:

To reproduce the failure I built kubeadm from the release-1.13 branch. I used that to build an etcd server. I created a control plane that had to resolve a dns lookup to connect to etcd. This dns lookup was defined in /etc/hosts, but the kube-apiserver was unable to resolve the etcd host.

I repeated the process using a kubeadm from the 1.12.2 release and the kube-apiserver was able to resolve and connect to etcd with no issue.

[anfernee]

what is the current solution for the case where apiserver wants to resolve hostname using kube-dns?