Feature request: HPA on controllers

[ericjohansson89]

Hello!

First I would like that we are using Flux more and more and think it is an excellent product.

We have started to come to a part where we are running Flux in multiple clusters with varying load in flux through the amount of Flux Custom Resources in the cluster. It feels like setting individual overrides per cluster for resources is an unwanted pattern. Rather I would like the controllers to scale horizontally based on the load.

The work could be delegated between the workers from the leader to split the load. I understand that the source controller might be hard to scale horizontally but all other controllers should be able to split the workload?

[kingdonb]

This is not at all straightforward to implement from today, as Flux controllers are not at present designed to be horizontally scaled.

They all implement the same leader election protocol as far as I understand, which means that no matter how you scale the deployments with an HPA, only one instance of a controller will be managing its associated CRD resources at any given time.

You can however set --concurrency to increase the number of resources that a Flux controller will process in parallel (and this comes with a performance penalty in the form of increased memory usage, mostly, for the controllers that handle objects which have a large memory footprint like Helm Controller and Source Controller, and probably Kustomize Controller as well.)

You will also need to increase the memory limit if you increased --concurrency. This is vertical scaling which is not what you asked for, but it is what Flux supports for now at least. Hope this helps!

[matheuscscp]

we are currently deciding whether we deploy Flux in-cluster for each cluster, or deploy a single Flux instance inside an infra cluster managing everything else in the non-infra clusters. if horizontal scaling is not supported yet then I think a single Flux will lead to problems in a couple of years according to our growth expectations

[kingdonb]

The --watch-all-namespaces flag is there, so you can use it to isolate multiple instances of Flux from one another, but I would need to see performance of what you're describing and check if the guidance we provide has any effect before I recommend that anti-pattern.

If your growth expectations are scaling to the point where a single Flux pod with high concurrency as the document about scaling describes, and you've tried this advice already:

https://fluxcd.io/docs/cheatsheets/bootstrap/#increase-the-number-of-workers
... and it isn't helping, then we can take a look at it.

But if your problem is purely hypothetical, I doubt that you will find a single Flux pod can't be scaled up to handle whatever your Kubernetes control plane can also handle. It's a shared bottleneck according to my understanding, so it seems more likely to me in your hypothetical scaling situation that you overloaded the Kubernetes control plane itself, rather than up past what Flux can handle.

Can you say more about the specific expectations, and whether you have any experimental data that suggests Flux is the bottleneck that can't be scaled past by simply adding more memory/CPU and increasing the concurrency as described in the doc? It's easy to make educated guesses, but hard data are a lot more actionable.

[matheuscscp]

actually we are at a very early stage with Flux and dont have any data yet, but we do have an important security decision for our current CD tool which is having it only in a separate infra cluster, and deploy all our apps across all our clusters from there. this cluster has more isolation, the other clusters cant initiate connections to it (only the other way around is allowed in the firewall), and this cluster is the only one with network connectivity to our git service at the moment.

we want to replace our current CD tool with Flux, so the first question is: can we keep this architecture/security decision?

I know that Flux supports this architecture with the obvious additional cost in operational complexity (configuring service accounts for external use in each cluster, etc.), which is probably a cost we are willing to pay. but how does it scale in terms of performance? if it turns out that sharding/horizontal scaling is not supported, I'm not sure I want to even try and see the scale limits for N apps x M clusters, and would rather recommend a different approach to my team from the beginning

[kingdonb]

I'm afraid we are tilting at windmills without data to support ourselves, and I've been somewhat confused about which thread this question came from and which problem it was aimed at solving. Sorry for any confusion. There are two different questions, and they don't necessarily have the same answer. Managing many instances of Flux on the same cluster is not generally supported, and if you're looking for guidance about how to build your Flux installation according to best practices, the advice is to scale with the concurrency model of Flux, which is not horizontal pod autoscalers, but adding more goroutines and enough memory/CPU to support them, for the service of additional concurrency.

You should not add more pods to Flux controllers because Flux will do leader election, and then only one controller pod will be effective while the others wait in "hot-standby" mode in case they are the next elected leader. If you want to run multiple instances of Flux on the same cluster disabling this leader election, that is not supported by Controller Runtime upstream and it seems unlikely to be supported soon. Flux would certainly not implement this support for sharding controllers ahead of Controller Runtime, which is a standard for Kubernetes controllers.

1. Can you run Flux on one cluster, multiple times, to shard and support different workloads independently on the same cluster. – Yes, you can, but it's not recommended to disable the leader election, and I wouldn't design my large installation of Flux this way. I do not have large swarths of thousand clusters with 10 Flux Kustomizations each to test with so I can't give specific performance numbers. I'm not sure at what point the scale would break down. The pull model is generally recommended instead, for another reason – so there is no single point of failure or "main cluster with access to every other cluster" that could be compromised.

This SPOF seems like a much riskier thing to have than allowing clusters to access the Git repo without going through a sort of "home office cluster" with a trusted/privileged position first. Installing Flux onto each cluster and letting the definition for what workloads to install on the cluster should live on that cluster is a generally better approach, and I think it resolves the scaling question that you had.

2. Can you manage many clusters from one central instance of Flux – yes, you can.

Using the Flux Kustomization.spec.kubeconfig, you can manage many Flux clusters from one central cluster, even prior to installing Flux on those target clusters. This is the pattern that we use in our enterprise product too: Cluster-API provides a cluster-admin secret that our (enterprise Weave GitOps) Cluster Bootstrap Controller uses to bootstrap another instance of Flux on the target cluster, along with some base configuration, usually to bootstrap the support for your cluster's RBAC at a minimum, (and whatever else).

Then from that point on there is no SPOF or "regular updates" that flow from any central cluster. The central cluster can go down, and updates continue to flow to the clusters without any centralized point of failure. This is neither in conflict with the idea of keeping central control nor the different idea of decentralized operation, both goals are met. An important factor for scaling is that now, heavy lifting can be done on the target clusters and each cluster serves as a natural boundary for the "sharding" of Flux. The main Flux cluster is only responsible for sending instructions to every target cluster, probably only flux-system and some Flux Kustomization resources there; and the remainder of the scale problem is solved naturally because each cluster has only its own local workloads to manage, and not some tree of many clusters.

Besides bootstrapping from the management cluster, each Flux handles the workload on its own cluster, and no Flux instance needs to wait very long to update itself, or to scale to outrageous proportion to manage 10k clusters. Once for a brief while (at the time they are first being bootstrapped) is likely the only time that our management cluster needs to operate on all 10k clusters at once. Even during upgrades, you can avoid the central bottleneck and push out updates to all clusters simultaneously, or in batches, whatever makes the most sense for you. The source of truth all comes from Git.

Since target clusters are not required to have any write access to Git, and at no point after bootstrap are the target clusters required to receive instructions from a central cluster; we think this is more secure as a pattern and more "GitOps-y". You can make it even more secure through Cosign verification and OCI artifacts; that way your releases are verified with some policy before they get sent out to the clusters, and the verification can be performed again in-situ on the clusters that are deploying the verified workloads.

[aristapimenta]

Can you run Flux on one cluster, multiple times, to shard and support different workloads independently on the same cluster. – Yes, you can

@kingdonb how exactly is this possible at the moment? what "multiple times" means exactly?

[stefanprodan]

Support for horizontal scaling of all Flux controllers with sharding based on label selectors is coming, see fluxcd/pkg#525