Use clusterresolver load balancer without full xDS support

[s-matyukevich]

Use case(s) - what problem will this feature solve?

Some of the advanced load-balancing features require better service-discovery protocols than DNS. For example, in order to implement zonal failover (or locality-aware load-balancing in general) we need to attach locality information to every endpoint. EDS allows to do that, however, currently it is not possible to consume just EDS without fully migrating our grpc clients to xDS. Full xDS migration is an option for us, but it is hard to do it in a granular way.

I created this issue to explore the possibility of using just EDS (via clusterresolver load balancer) without full migration to xDS. We already did something similar for outlierdetection load balancer #5899

Proposed Solution

Users should be able to set ServiceConfig similar to this one, and configure clusterresolver LB themselves.

{
  "loadBalancingConfig": [
    {
       "cluster_resolver_experimental": {
         "edsServiceName": <some EDS cluster name>,
         "type": "EDS",
         "outlierDetection": <optional outlier-detection config>
        }
    }
  ]
}

I created a small PoC to test this, and the only blocker I found was the fact that clusterresolver LB expects parent LB to get xdsclient from resolver state (it happens here) Here are 2 possible options how we can deal with it:

• Move XDSClient interface out of internal package, so we can create our own instance of this interface and pass it to the clusterresolver LB.
• Modify clusterresolver LB to create a new instance of the XDSClient in case when nothing is found in the resolver state.

Alternatives Considered

Another option we are considering is to write an xDS management server which will basically respond with a hardcoded set of xDS resources (listeners and clusters) with the exception that EDS resource reference will be generated based on the requested cluster name or based on some request metadata. With such a server we can migrate all our grpc clients to xDS.

Additional Context

[arvindbr8]

Hey! Nice to see you again @s-matyukevich.

Here are 2 possible options how we can deal with it:

Move XDSClient interface out of internal package, so we can create our own instance of this interface and pass it to the clusterresolver LB.

We do not want to expose xDSClient interface by moving it out of internal package. Also seems like we might have to expose other APIs from internal.

Modify clusterresolver LB to create a new instance of the XDSClient in case when nothing is found in the resolver state.

This is totally possible, but does not seem like a good idea to add it to gRPC-go. You could however implement a custom clusterresolver LB in your fork.

Full xDS migration is an option for us, but it is hard to do it in a granular way.

Could you elaborate why implementing a hardcoded set of xDS resources (for Listener and clusters) is considered a hard option?

We feel that setting up a management server to respond with hardcoded LDS/CDS/RDS resources seems like the easiest approach here (unless we are missing something here - hence requires clarification). You could think about using go control plane management server implementation to check this out.

---

Meanwhile, we are doing some digging from our end to see if gRPC should support an EDS only flow.

[s-matyukevich]

Could you elaborate why implementing a hardcoded set of xDS resources (for Listener and clusters) is considered a hard option?

Some of our users care a lot about internal complexity of the libraries we use, which could bring some entirely new failure modes. Adding xDS to the picture increases complexity a lot. In case of EDS it is easy to explain why do we need it, and users can look at EDS as a more advanced DNS version, so it is likely they won't push-back hard on adopting it. In case of xDS with hardcoded resources we get a big internal complexity increase with no added value, which is hard to sell.

One selling point that we are trying to use is the fact that if we switch to xDS this would make it easier to add dynamic behaviors to grpc clients in the future. This allows service owners to automatically configure some aspects of their clients. However, one major issue with that is the fact that grpc xDS support is very limited and doesn't allow configuring things like transport-level settings, healthchecks, etc. So if our users will use some centralized system to configure things like retries and outlier detection parameters, but keep configuring things like keepalives and max message limits in the client code, IMO it will be inconsistent and bad user experience.

[arvindbr8]

All implementations of gRPC currently does not support an EDS only xDS flow. It doesn't seem like something we want to support it at the moment because most of our users have a complete xDS flow.

I also believe that easiest and the most straightforward option here would be if you could set up the management server to respond with hardcoded LDS, CDS, and RDS responses.

Closing this feature request. Please reopen if you have any further questions.