clientv3: refresh authToken when permission deny

[jschwinger233]

clientv3: bring watch interceptor to re-fetch auth token

watch interceptor will work only when ClientStream.RecvMsg fulfill 2 requirements:

1. last sent message was WatchCreateRequest
2. this received message indicates permission deny

on other situations the interceptor is transparent and insignificant.

fix:
#11954
#11121
#11381

[jschwinger233]

CI failed, what shoud I do to get pass? I can't even pass docker run --rm --volume=pwd:/go/src/go.etcd.io/etcd gcr.io/etcd-development/etcd-test:go1.14.3 /bin/bash -c "GOARCH=amd64 CPU=1 PASSES='integration' ./test" on upstream master HEAD.

[tangcong]

etcd supports multiple authentication methods. If users use other authentication methods, such as certificates. have you tested this scenario(cert has no permission)?
It seems that we need to distinguish token authentication. for example, in unaryClientInterceptor.

			if callOpts.retryAuth && rpctypes.Error(lastErr) == rpctypes.ErrInvalidAuthToken {
				gterr := c.getToken(ctx)
				if gterr != nil {
					logger.Warn(
						"retrying of unary invoker failed to fetch new auth token",
						zap.String("target", cc.Target()),
						zap.Error(gterr),
					)
					return gterr // lastErr must be invalid auth token
				}
				continue
			}

[mitake]

In the case of CN based auth, users won't supply username and password, then getToken() will be nop so it will be fine. But caring about different auth mechanisms are good point.

@jschwinger23 could you add e2e test cases for making sure this refreshing mechanism works and it won't break other auth mechanisms? If you are busy adding in other PRs is fine.

[tangcong]

It seems that it cannot distinguish different auth mechanisms here. I agree with @xkeyideal solution in issue #12157.

[tangcong]

/cc @mitake can you take a look. thanks.

[mitake]

LGTM, thanks! Defer to @gyuho @jingyih or other maintainers.

[xiang90]

remove empty line here.

[jschwinger233]

fixed by #12264, close