Add IDestinationResolver for resolving cluster destination addresses

docs/docfx/articles/destination-resolvers.md

@@ -0,0 +1,58 @@
+# Extensibility: Destination Resolvers
+
+## Introduction
+
+YARP uses a destination resolver to expand the set of configured destination addresses. The destination resolver can be used as an integration point with service discovery systems.
+
+## Structure
+[IDestinationResolver](xref:Yarp.ReverseProxy.ServiceDiscovery.IDestinationResolver) has a single method `ResolveDestinationsAsync(IReadOnlyDictionary<string, DestinationConfig> destinations, CancellationToken cancellationToken)` which should return a [ResolvedDestinationCollection](xref:Yarp.ReverseProxy.ServiceDiscovery.ResolvedDestinationCollection) instance. The [ResolvedDestinationCollection](xref:Yarp.ReverseProxy.ServiceDiscovery.ResolvedDestinationCollection) has a collection of [DestinationConfig](xref:Yarp.ReverseProxy.Configuration.DestinationConfig) instances, as well as an `IChangeToken` to notify the proxy when this information is out of date and should be reloaded, which will cause `ResolveDestinationsAsync` to be called again.
+
+### DestinationConfig
+`DestinationConfig` has a `Host` property which can be used to specify the default `Host` header value which the proxy should use when communicating with that destination. This allows the `IDestinationResolver` to resolve destinations to a collection of IP addresses, for example, without causing SNI or host-based routing to fail.
+
+## Lifecycle
+
+### Startup
+The `IDestinationResolver` should be registered in the DI container as a singleton. At startup, the proxy will resolve this instance and call `ResolveDestinationsAsync(...)` with the configured destinations retrieved from the resolved `IProxyConfigProviders`. On this first call the provider may choose to:
+- Throw an exception if the provider cannot produce a valid proxy configuration for any reason. This will prevent the application from starting.
+- Asynchronously resolve the destinations. This will stop the application from starting until resolved destinations are available.
+- Or, it may choose to return an empty `ResolvedDestinationCollection` instance while it resolves destinations in the background. The provider will need to trigger the `IChangeToken` when the configuration is available.
+
+### Atomicity
+The destinations objects and collections supplied to the proxy should be read-only and not modified once they have been handed to the proxy via `GetConfig()`.
+
+### Reload
+If the `IChangeToken` supports `ActiveChangeCallbacks`, once the proxy has processed the initial set of destinations it will register a callback with this token. If the provider does not support callbacks then `HasChanged` will be polled alongside `IProxyConfig` change tokens, every 5 minutes.
+
+When the provider wants to provide a new set of destinations to the proxy it should:
+- Resolve those destinations in the background.
+  - `ResolvedDestinationCollection` is immutable, so new instances have to be created for any new data.
+  - Objects for unchanged destinations can be re-used, or new instances can be created.
+- Invalidate the `IChangeToken` returned from the previous `ResolveDestinationsAsync` invocation.
+
+Once the new destinations have been applied, the proxy will register a callback with the new `IChangeToken`. Note if there are multiple reloads signaled in close succession, the proxy may skip some and resolve destinations as soon as it's ready.
+
+## DNS Destination Resolver
+
+YARP includes an [IDestinationResolver](xref:Yarp.ReverseProxy.ServiceDiscovery.IDestinationResolver) implementation which expands the set of configured destinations by resolving each host name to one or more IP addresses using DNS, creating a destination for each resolved IP.
+The DNS destination resolver can be added to your reverse proxy using the `IReverseProxyBuilder.AddDnsDestinationResolver(Action<DnsDestinationResolverOptions>)` method.
+The method accepts an optional delegate to configure the resolver's options, [DnsDestinationResolverOptions](xref:Yarp.ReverseProxy.ServiceDiscovery.DnsDestinationResolverOptions).
+
+### Example
+
+```csharp
+// Add the DNS destination resolver, restricting results to IPv4 addresses
+reverseProxyBuilder.AddDnsDestinationResolver(o => o.AddressFamily = AddressFamily.InterNetwork);
+```
+
+### Configuration
+
+The DNS destination resolver's options, [DnsDestinationResolverOptions](xref:Yarp.ReverseProxy.ServiceDiscovery.DnsDestinationResolverOptions), has the following properties:
+
+#### RefreshPeriod
+
+The period between requesting a refresh of a resolved name. This defaults to 5 minutes.
+
+#### AddressFamily
+
+Optionally, specify an `System.Net.Sockets.AddressFamily` value of `AddressFamily.InterNetwork` or `AddressFamily.InterNetworkV6` to restrict resolved resolution to IPv4 or IPv6 addresses, respectively. The default value, `null`, instructs the resolver to not restrict the address family of the results and to use accept all returned addresses.

docs/docfx/articles/toc.yml

@@ -10,6 +10,8 @@
   href: config-filters.md
 - name: Direct Forwarding
   href: direct-forwarding.md
+- name: Destination Resolvers
+  href: destination-resolvers.md
 - name: HTTP client configuration
   href: http-client-config.md
 - name: HTTPS & TLS

src/ReverseProxy/Configuration/ConfigProvider/ConfigurationConfigProvider.cs

@@ -377,6 +377,7 @@ private static DestinationConfig CreateDestination(IConfigurationSection section
             Address = section[nameof(DestinationConfig.Address)]!,
             Health = section[nameof(DestinationConfig.Health)],
             Metadata = section.GetSection(nameof(DestinationConfig.Metadata)).ReadStringDictionary(),
+            Host = section[nameof(DestinationConfig.Host)] 
         };
     }
 

src/ReverseProxy/Configuration/DestinationConfig.cs

@@ -28,6 +28,12 @@ public sealed record DestinationConfig
     /// </summary>
     public IReadOnlyDictionary<string, string>? Metadata { get; init; }
 
+    /// <summary>
+    /// Host header value to pass to this destination.
+    /// Used as a fallback if a host is not already specified by request transforms.
+    /// </summary>
+    public string? Host { get; init; }
+
     public bool Equals(DestinationConfig? other)
     {
         if (other is null)
@@ -37,6 +43,7 @@ public bool Equals(DestinationConfig? other)
 
         return string.Equals(Address, other.Address, StringComparison.OrdinalIgnoreCase)
             && string.Equals(Health, other.Health, StringComparison.OrdinalIgnoreCase)
+            && string.Equals(Host, other.Host, StringComparison.OrdinalIgnoreCase)
             && CaseSensitiveEqualHelper.Equals(Metadata, other.Metadata);
     }
 
@@ -45,6 +52,7 @@ public override int GetHashCode()
         return HashCode.Combine(
             Address?.GetHashCode(StringComparison.OrdinalIgnoreCase),
             Health?.GetHashCode(StringComparison.OrdinalIgnoreCase),
+            Host?.GetHashCode(StringComparison.OrdinalIgnoreCase),
             CaseSensitiveEqualHelper.GetHashCode(Metadata));
     }
 }

src/ReverseProxy/Health/DefaultProbingRequestFactory.cs

@@ -28,6 +28,11 @@ public HttpRequestMessage CreateRequest(ClusterModel cluster, DestinationModel d
             VersionPolicy = cluster.Config.HttpRequest?.VersionPolicy ?? HttpVersionPolicy.RequestVersionOrLower,
         };
 
+        if (!string.IsNullOrEmpty(destination.Config.Host))
+        {
+            request.Headers.Add(HeaderNames.Host, destination.Config.Host);
+        }
+
         request.Headers.Add(HeaderNames.UserAgent, _defaultUserAgent);
 
         return request;

src/ReverseProxy/Management/IReverseProxyBuilderExtensions.cs

@@ -12,6 +12,7 @@
 using Yarp.ReverseProxy.LoadBalancing;
 using Yarp.ReverseProxy.Model;
 using Yarp.ReverseProxy.Routing;
+using Yarp.ReverseProxy.ServiceDiscovery;
 using Yarp.ReverseProxy.SessionAffinity;
 using Yarp.ReverseProxy.Transforms;
 using Yarp.ReverseProxy.Utilities;
@@ -125,4 +126,10 @@ public static IReverseProxyBuilder AddHttpSysDelegation(this IReverseProxyBuilde
 
         return builder;
     }
+
+    public static IReverseProxyBuilder AddDestinationResolver(this IReverseProxyBuilder builder)
+    {
+        builder.Services.TryAddSingleton<IDestinationResolver, NoOpDestinationResolver>();
+        return builder;
+    }
 }

src/ReverseProxy/Management/ProxyConfigManager.cs

@@ -20,6 +20,7 @@
 using Yarp.ReverseProxy.Health;
 using Yarp.ReverseProxy.Model;
 using Yarp.ReverseProxy.Routing;
+using Yarp.ReverseProxy.ServiceDiscovery;
 using Yarp.ReverseProxy.Transforms.Builder;
 
 namespace Yarp.ReverseProxy.Management;
@@ -49,6 +50,7 @@ internal sealed class ProxyConfigManager : EndpointDataSource, IProxyStateLookup
     private readonly List<Action<EndpointBuilder>> _conventions;
     private readonly IActiveHealthCheckMonitor _activeHealthCheckMonitor;
     private readonly IClusterDestinationsUpdater _clusterDestinationsUpdater;
+    private readonly IDestinationResolver _destinationResolver;
     private readonly IConfigChangeListener[] _configChangeListeners;
     private List<Endpoint>? _endpoints;
     private CancellationTokenSource _endpointsChangeSource = new();
@@ -67,7 +69,8 @@ public ProxyConfigManager(
         IForwarderHttpClientFactory httpClientFactory,
         IActiveHealthCheckMonitor activeHealthCheckMonitor,
         IClusterDestinationsUpdater clusterDestinationsUpdater,
-        IEnumerable<IConfigChangeListener> configChangeListeners)
+        IEnumerable<IConfigChangeListener> configChangeListeners,
+        IDestinationResolver destinationResolver)
     {
         _logger = logger ?? throw new ArgumentNullException(nameof(logger));
         _providers = providers?.ToArray() ?? throw new ArgumentNullException(nameof(providers));
@@ -80,7 +83,7 @@ public ProxyConfigManager(
         _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
         _activeHealthCheckMonitor = activeHealthCheckMonitor ?? throw new ArgumentNullException(nameof(activeHealthCheckMonitor));
         _clusterDestinationsUpdater = clusterDestinationsUpdater ?? throw new ArgumentNullException(nameof(clusterDestinationsUpdater));
-
+        _destinationResolver = destinationResolver ?? throw new ArgumentNullException(nameof(destinationResolver));
         _configChangeListeners = configChangeListeners?.ToArray() ?? Array.Empty<IConfigChangeListener>();
 
         if (_providers.Length == 0)
@@ -158,11 +161,19 @@ internal async Task<EndpointDataSource> InitialLoadAsync()
             var routes = new List<RouteConfig>();
             var clusters = new List<ClusterConfig>();
 
+            // Begin resolving config providers concurrently.
+            var resolvedConfigs = new List<(int Index, IProxyConfigProvider Provider, ValueTask<IProxyConfig> Config)>(_providers.Length);
             for (var i = 0; i < _providers.Length; i++)
             {
                 var provider = _providers[i];
-                var config = provider.GetConfig();
-                ValidateConfigProperties(config);
+                var configLoadTask = LoadConfigAsync(provider, cancellationToken: default);
+                resolvedConfigs.Add((i, provider, configLoadTask));
+            }
+
+            // Wait for all configs to be resolved.
+            foreach (var (i, provider, configLoadTask) in resolvedConfigs)
+            {
+                var config = await configLoadTask;
                 _configs[i] = new ConfigState(provider, config);
                 routes.AddRange(config.Routes ?? Array.Empty<RouteConfig>());
                 clusters.AddRange(config.Clusters ?? Array.Empty<ClusterConfig>());
@@ -202,33 +213,52 @@ private async Task ReloadConfigAsync()
         var sourcesChanged = false;
         var routes = new List<RouteConfig>();
         var clusters = new List<ClusterConfig>();
+        var reloadedConfigs = new List<(ConfigState Config, ValueTask<IProxyConfig> ResolveTask)>();
+
+        // Start reloading changed configurations.
         foreach (var instance in _configs)
         {
-            try
+            if (instance.LatestConfig.ChangeToken.HasChanged)
             {
-                if (instance.LatestConfig.ChangeToken.HasChanged)
+                try
+                {
+                    var reloadTask = LoadConfigAsync(instance.Provider, cancellationToken: default);
+                    reloadedConfigs.Add((instance, reloadTask));
+                }
+                catch (Exception ex)
                 {
-                    var config = instance.Provider.GetConfig();
-                    ValidateConfigProperties(config);
-                    instance.LatestConfig = config;
-                    instance.LoadFailed = false;
-                    sourcesChanged = true;
+                    OnConfigLoadError(instance, ex);
                 }
             }
+        }
+
+        // Wait for all changed config providers to be reloaded.
+        foreach (var (instance, loadTask) in reloadedConfigs)
+        {
+            try
+            {
+                instance.LatestConfig = await loadTask.ConfigureAwait(false);
+                instance.LoadFailed = false;
+                sourcesChanged = true;
+            }
             catch (Exception ex)
             {
-                instance.LoadFailed = true;
-                Log.ErrorReloadingConfig(_logger, ex);
+                OnConfigLoadError(instance, ex);
+            }
+        }
 
-                foreach (var configChangeListener in _configChangeListeners)
-                {
-                    configChangeListener.ConfigurationLoadingFailed(instance.Provider, ex);
-                }
+        // Extract the routes and clusters from the configs, regardless of whether they were reloaded.
+        foreach (var instance in _configs)
+        {
+            if (instance.LatestConfig.Routes is { Count: > 0 } updatedRoutes)
+            {
+                routes.AddRange(updatedRoutes);
             }
 
-            // If we didn't/couldn't get a new config then re-use the last one.
-            routes.AddRange(instance.LatestConfig.Routes ?? Array.Empty<RouteConfig>());
-            clusters.AddRange(instance.LatestConfig.Clusters ?? Array.Empty<ClusterConfig>());
+            if (instance.LatestConfig.Clusters is { Count: > 0 } updatedClusters)
+            {
+                clusters.AddRange(updatedClusters);
+            }
         }
 
         var proxyConfigs = ExtractListOfProxyConfigs(_configs);
@@ -270,6 +300,17 @@ private async Task ReloadConfigAsync()
         }
 
         ListenForConfigChanges();
+
+        void OnConfigLoadError(ConfigState instance, Exception ex)
+        {
+            instance.LoadFailed = true;
+            Log.ErrorReloadingConfig(_logger, ex);
+
+            foreach (var configChangeListener in _configChangeListeners)
+            {
+                configChangeListener.ConfigurationLoadingFailed(instance.Provider, ex);
+            }
+        }
     }
 
     private static void ValidateConfigProperties(IProxyConfig config)
@@ -278,12 +319,90 @@ private static void ValidateConfigProperties(IProxyConfig config)
         {
             throw new InvalidOperationException($"{nameof(IProxyConfigProvider.GetConfig)} returned a null value.");
         }
+
         if (config.ChangeToken is null)
         {
             throw new InvalidOperationException($"{nameof(IProxyConfig.ChangeToken)} has a null value.");
         }
     }
 
+    private ValueTask<IProxyConfig> LoadConfigAsync(IProxyConfigProvider provider, CancellationToken cancellationToken)
+    {
+        var config = provider.GetConfig();
+        ValidateConfigProperties(config);
+
+        if (_destinationResolver.GetType() == typeof(NoOpDestinationResolver))
+        {
+            return new(config);
+        }
+
+        return LoadConfigAsyncCore(config, cancellationToken);
+    }
+
+    private async ValueTask<IProxyConfig> LoadConfigAsyncCore(IProxyConfig config, CancellationToken cancellationToken)
+    {
+        List<(int Index, ValueTask<ResolvedDestinationCollection> Task)> resolverTasks = new();
+        List<ClusterConfig> clusters = new(config.Clusters);
+        List<IChangeToken>? changeTokens = null;
+        for (var i = 0; i < clusters.Count; i++)
+        {
+            var cluster = clusters[i];
+            if (cluster.Destinations is { Count: > 0 } destinations)
+            {
+                // Resolve destinations if there are any.
+                var task = _destinationResolver.ResolveDestinationsAsync(destinations, cancellationToken);
+                resolverTasks.Add((i, task));
+            }
+        }
+
+        if (resolverTasks.Count > 0)
+        {
+            foreach (var (i, task) in resolverTasks)
+            {
+                var resolvedDestinations = await task;
+                clusters[i] = clusters[i] with { Destinations = resolvedDestinations.Destinations };
+                if (resolvedDestinations.ChangeToken is { } token)
+                {
+                    changeTokens ??= new();
+                    changeTokens.Add(token);
+                }
+            }
+
+            IChangeToken changeToken;
+            if (changeTokens is not null)
+            {
+                // Combine change tokens from the resolver with the configuration's existing change token.
+                changeTokens.Add(config.ChangeToken);
+                changeToken = new CompositeChangeToken(changeTokens);
+            }
+            else
+            {
+                changeToken = config.ChangeToken;
+            }
+
+            // Return updated config
+            return new ResolvedProxyConfig(config, clusters, changeToken);
+        }
+
+        return config;
+    }
+
+    private sealed class ResolvedProxyConfig : IProxyConfig
+    {
+        private readonly IProxyConfig _innerConfig;
+
+        public ResolvedProxyConfig(IProxyConfig innerConfig, IReadOnlyList<ClusterConfig> clusters, IChangeToken changeToken)
+        {
+            _innerConfig = innerConfig;
+            Clusters = clusters;
+            ChangeToken = changeToken;
+        }
+
+        public IReadOnlyList<RouteConfig> Routes => _innerConfig.Routes;
+        public IReadOnlyList<ClusterConfig> Clusters { get; }
+        public IChangeToken ChangeToken { get; }
+    }
+
     private void ListenForConfigChanges()
     {
         // Use a central change token to avoid overlap between different sources.