Add controller sharding capability based on label selector

[stefanprodan]

Add --watch-label-selector=<selector expression> flag to all Flux controller using the new watch package from fluxcd/pkg/runtime and wired the selector in each controller's cache manager.

This enables horizontal scaling of Flux controllers, where each controller can be deployed multiple times with a unique label selector which is used as the sharding key.

For example, to shard the reconciliation of HelmReleases in a cluster, user will deploy two extra helm-controller instances:

• Deployment/helm-controller with --watch-label-selector=sharding.fluxcd.io/shard notin (shard1, shard2)
• Deployment/helm-controller-1 with --watch-label-selector=sharding.fluxcd.io/shard=shard1
• Deployment/helm-controller-2 with --watch-label-selector=sharding.fluxcd.io/shard=shard2

By labeling HelmReleases with sharding.fluxcd.io/shard, users can spread the load between helm-controller-1 and helm-controller-2 by setting the label value to shard1 or shard2. HelmReleases which don't have the sharding.fluxcd.io/shard label, will be reconcile by the helm-controller default instance.

Sharding source-controller works the same but users will have to create a matching Kubernetes Service for each SC deployment, then set the Service name in storage address flag. Example:

• Deployment+Service source-controller-1 with:

--storage-adv-addr=source-controller-1.$(RUNTIME_NAMESPACE).svc.cluster.local.
--watch-label-selector=sharding.fluxcd.io/shard=shard1

• Deployment+Service source-controller-2 with:

--storage-adv-addr=source-controller-2.$(RUNTIME_NAMESPACE).svc.cluster.local.
--watch-label-selector=sharding.fluxcd.io/shard=shard12

Tasks:

• fluxcd/source-controller
• fluxcd/helm-controller
• fluxcd/kustomize-controller
• Add reconciler sharding capability based on label selector image-reflector-controller#364
• Add reconciler sharding capability based on label selector image-automation-controller#503