Driver registrar can sometimes take 1 minute to start

[saad-ali]

Driver registrar can sometimes take 1 minute to start. See kubernetes/kubernetes#71433 (comment)

Let's understand why, and see if we can reduce this.

[saad-ali]

Relevant code:

driver-registrar/pkg/connection/connection.go

Lines 86 to 96 in 0c989a8

	for {
	if !conn.WaitForStateChange(ctx, conn.GetState()) {
	glog.V(4).Infof("Connection timed out")
	return conn, nil // return nil, subsequent GetPluginInfo will show the real connection error
	}
	if conn.GetState() == connectivity.Ready {
	glog.V(3).Infof("Connected")
	return conn, nil
	}
	glog.V(4).Infof("Still trying, connection is %s", conn.GetState())
	}

In this case it received TRANSIENT_FAILURE. Apparently doing nothing in that case doesn't fix anything.

[pohly]

Something that mitigates the issue is to put the CSI driver container first in the list of containers for the pod. I have seen timing related delays on a slow cluster because Kubernetes pulled images one-by-one, then started registrar first before it started pulling the CSI driver image.

But the real fix will be to make that connect loop iterate faster. This is code that should go into the new kubernetes-csi/csi-utils repo that we started to discuss for gRPC logging.

[msau42]

Another suggestion is to try different rpc options: kubernetes/kubernetes#71433 (comment)

[vladimirvivien]

Investigating a fix/workaround based on suggestions here.

[vladimirvivien]

Created PR - #78

[lpabon]

🛂 ⛔️
Hi, this repo has been split up into two repos:

• cluster-driver-registrar: Registers the driver with Kubernetes by setting up a CRD
• node-driver-registrar: Registers the driver with the kubelet on the node.

This repo has been closed, and no new changes will be accepted. Please move your content to one of these repos.

Thank you,