Bug 1698456 *: use wildcard domain in DNS: SAN for etcd server certs

[hexfusion]

This PR resolves an issue with client balancer and etcd. The balancer is populated with a list of etcd peer endpoints. When we dial endpoint[0] it is used as the target and the other endpoints are dialed using subconnections. I have verified that each connection, the target and subs all make a proper TLS handshake with Wireshark.

The issue we see which is painted well in the below logs, when etcd-0 fails and the balancer failsover to etcd-1 the connection will fail because the TLS context of the balancer assumes target (etcd-0).

1 clientconn.go:1304] grpc: addrConn.createTransport failed to connect to {etcd-1.jliu-demo.qe.devcluster.openshift.com:2379 0 }. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, etcd.kube-system.svc, etcd.kube-system.svc.cluster.local, etcd-1.jliu-demo.qe.devcluster.openshift.com, not etcd-0.jliu-demo.qe.devcluster.openshift.com". Reconnecting...

The solution, for now, is to populate the DNS: SAN of server certs with a wildcard. This will allow TLS auth to complete successfully and the balancer can properly work. This is because the target etcd-0 will now authenticate against the *.clustername.domain.com in SAN.

Check openshift-kube-apiserver logs
I0423 06:53:53.612060       1 resolver_conn_wrapper.go:116] ccResolverWrapper: sending new addresses to cc: [{etcd-0.jliu-demo.qe.devcluster.openshift.com:2379 0  <nil>}]
I0423 06:53:53.612145       1 balancer_v1_wrapper.go:125] balancerWrapper: got update addr from Notify: [{etcd-0.jliu-demo.qe.devcluster.openshift.com:2379 <nil>} {etcd-1.jliu-demo.qe.devcluster.openshift.com:2379 <nil>} {etcd-2.jliu-demo.qe.devcluster.openshift.com:2379 <nil>}]
W0423 06:53:53.654818       1 clientconn.go:1304] grpc: addrConn.createTransport failed to connect to {etcd-1.jliu-demo.qe.devcluster.openshift.com:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, etcd.kube-system.svc, etcd.kube-system.svc.cluster.local, etcd-1.jliu-demo.qe.devcluster.openshift.com, not etcd-0.jliu-demo.qe.devcluster.openshift.com". Reconnecting...
I0423 06:53:53.660209       1 balancer_v1_wrapper.go:125] balancerWrapper: got update addr from Notify: [{etcd-0.jliu-demo.qe.devcluster.openshift.com:2379 <nil>}]
W0423 06:53:53.660243       1 clientconn.go:953] Failed to dial etcd-1.jliu-demo.qe.devcluster.openshift.com:2379: context canceled; please retry.
I0423 06:53:53.686866       1 master.go:228] Using reconciler: lease
I0423 06:53:53.687309       1 clientconn.go:551] parsed scheme: ""
I0423 06:53:53.687324       1 clientconn.go:557] scheme "" not registered, fallback to default scheme
I0423 06:53:53.687354       1 resolver_conn_wrapper.go:116] ccResolverWrapper: sending new addresses to cc: [{etcd-0.jliu-demo.qe.devcluster.openshift.com:2379 0  <nil>}]
I0423 06:53:53.687387       1 balancer_v1_wrapper.go:125] balancerWrapper: got update addr from Notify: [{etcd-0.jliu-demo.qe.devcluster.openshift.com:2379 <nil>} {etcd-1.jliu-demo.qe.devcluster.openshift.com:2379 <nil>} {etcd-2.jliu-demo.qe.devcluster.openshift.com:2379 <nil>}]
W0423 06:53:53.688678       1 clientconn.go:953] Failed to dial etcd-2.jliu-demo.qe.devcluster.openshift.com:2379: grpc: the connection is closing; please retry.
W0423 06:53:53.693322       1 clientconn.go:1304] grpc: addrConn.createTransport failed to connect to {etcd-1.jliu-demo.qe.devcluster.openshift.com:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, etcd.kube-system.svc, etcd.kube-system.svc.cluster.local, etcd-1.jliu-demo.qe.devcluster.openshift.com, not etcd-0.jliu-demo.qe.devcluster.openshift.com". Reconnecting...
W0423 06:53:53.693536       1 clientconn.go:1304] grpc: addrConn.createTransport failed to connect to {etcd-2.jliu-demo.qe.devcluster.openshift.com:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for localhost, etcd.kube-system.svc, etcd.kube-system.svc.cluster.local, etcd-2.jliu-demo.qe.devcluster.openshift.com, not etcd-0.jliu-demo.qe.devcluster.openshift.com". Reconnecting...
I0423 06:53:53.733572       1 balancer_v1_wrapper.go:125] balancerWrapper: got update addr from Notify: [{etcd-0.jliu-demo.qe.devcluster.openshift.com:2379 <nil>}]
W0423 06:53:53.733612       1 clientconn.go:953] Failed to dial etcd-1.jliu-demo.qe.devcluster.openshift.com:2379: context canceled; please retry.
W0423 06:53:53.733621       1 clientconn.go:953] Failed to dial etcd-2.jliu-demo.qe.devcluster.openshift.com:2379: context canceled; please retry.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1698456

Ref:

• etcd go client fails when querying a cluster with a down node etcd-io/etcd#9949
• [gRPC] clientconn: set authority with the latest dial target etcd-io/etcd#10489
• kube-apiserver 1.13.x refuses to work when first etcd-server is not available. kubernetes/kubernetes#72102
• clientconn: set dial target "Authority" with target address grpc/grpc-go#2650

/hold

[hexfusion]

/cc @deads2k @abhinavdahiya @smarterclayton

[hexfusion]

/cc @ericavonb

[smarterclayton]

This is reasonable to me, and the failure scenario is bad.

/approve

for 4.1 (the code looks correct to me, but want others to review deeper)

[smarterclayton]

As a side note, we should make sure you are in OWNERS under cmd/setup-etcd-environment and a few other dirs.

[runcom]

/approve

from mco point on view, will leave to others as well

[smarterclayton]

This lgtm, and I've heard no comment, and this is a huge blocker

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hexfusion, runcom, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[smarterclayton]

/hold cancel

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[smarterclayton]

/retest

[hexfusion]

level=error msg="1 error occurred:"
level=error msg="\t* module.vpc.aws_route.to_nat_gw[1]: 1 error occurred:"
level=error msg="\t* aws_route.to_nat_gw.1: Error creating route: timeout while waiting for state to become 'success' (timeout: 20m0s)"

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[hexfusion]

/test e2e-aws

[hexfusion]

/test e2e-aws

[hexfusion]

level=error msg="1 error occurred:"
level=error msg="\t* module.vpc.aws_route_table_association.route_net[1]: 1 error occurred:"
level=error msg="\t* aws_route_table_association.route_net.1: timeout while waiting for state to become 'success' (timeout: 5m0s)"

/test e2e-aws