etcd v2 to v3 migration instructions cause cluster downtime when using CoreDNS

[bohde]

1. What kops version are you running? The command kops version, will display
this information.

1.12.0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.8", GitCommit:"a89f8c11a5f4f132503edbc4918c98518fd504e3", GitTreeState:"clean", BuildDate:"2019-04-23T04:41:47Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

I attempted to update a 3 master setup using CoreDNS 1.3.1 and etcd v2.2.1 to v3.2.24, using the method documented in the etcd v3 migration guide.:

# Roll masters as quickly as possible
kops rolling-update cluster --cloudonly --instance-group-roles master --master-interval=1s
kops rolling-update cluster --cloudonly --instance-group-roles master --master-interval=1s --yes

5. What happened after the commands executed?

Whenever the master nodes go, all CoreDNS pods crash with the following message:

log: exiting because of error: log: cannot create log: open /tmp/coredns.coredns-7fd9fbf6cc-gk9ht.unknownuser.log.ERROR.20190524-195653.1: no such file or directory

They reboot, and connect to the new masters, but DNS is down in the cluster for approximately 1 minute. This issue is fixed in this CoreDNS pull request coredns/coredns#2529

6. What did you expect to happen?

No DNS downtime while the masters are down.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

  channel: stable
  cloudProvider: aws
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    - instanceGroup: master-us-west-2b
      name: b
    - instanceGroup: master-us-west-2c
      name: c
    name: main
    provider: Manager
    version: 3.2.24
  - etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    - instanceGroup: master-us-west-2b
      name: b
    - instanceGroup: master-us-west-2c
      name: c
    name: events
    provider: Manager
    version: 3.2.24
  kubeDNS:
    provider: CoreDNS
  kubernetesVersion: 1.12.8

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

Downgrading the CoreDNS deployment to 1.3.0 does not exhibit this problem.

[vivekgarg20]

We also faced same problem. Seems CoreDNS is dependent on API Server to be up.
coredns/coredns#2629.

We tested with 1.5.0 and problem is gone.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.