clientv3: customize/improve balancer endpoint selection

[gyuho]

Problem

Current balancer randomly pins first connected address and never switches until disconnected. Theoretically, millions of clients can be pinned onto the same node. Worse if several watchers are multiplexed on those connections--dispatching node can quickly be overloaded. Since we do not expose connection status or pinned endpoint, client has no way to distribute connections evenly.

Solution

v2 client has EndpointSelectionPrioritizeLeader. Similar can happen for v3 client. Better if we can let client specify dial policy: weighted roundrobin, prioritize leader, customized endpoint selection policy based on workloads, dynamic endpoint selection/switch based on workloads.

Related

• Expose endpoint connection status clientv3: expose endpoint connection status #8776
• Rewrite client balancer Rewrite clientv3 health balancer #9106

Probably after we implement new client balancer.

[gyuho]

Related #7683.