enhancement request: allow the option to specify kubectl to replace instead of apply

[MxNxPx]

Issue:
I'm running into a problem when using flux2 to generate & apply a large configmap (specifically a grafana dashboard) that it fails due to it stuffing the entire contents into the annotation: "kubectl.kubernetes.io/last-applied-configuration" and exceeding the character limit and throwing this error...
The ConfigMap "grafana-dashboard-3" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

(Same as this issue reported: #1301)

Findings:
From what I gather, flux2 is strictly doing a kubectl apply which even with the --force=true option still throws the error on the annotation. But when specifying kubectl replace --force=true the configmaps are deleted and then created successfully.

Proposal:
Could flux2 expose some flag whereby a particular Kustomization could specify the kubectl behavior? Namely in this case an "apply" versus "replace" perhaps?

Additional Info:
I put some details on recreating the issue using kind+flux2; and the behavior of kustomize & kubectl in my example repo here:
https://github.com/MxNxPx/flux2-issue

[dholbach]

Are the answers in #1301 (comment) and #1301 (comment) helpful?

As it see seems to be a etcd limit, is this something that should be documented in the FAQ maybe or what would be your expectation?

[Frizlab]

I don’t want to speak in the name of @MxNxPx, but I think the idea would be to allow flux to basically use kubectl create instead of kubectl apply for some objects (maybe add a new property on the Kustomization object to tell that objects should be “created” instead of “applied”?)

Going full GitOps, I know I don’t manipulate the cluster manually at all anymore. These kind of objects, which are too big to be “applied” are still possible in a cluster, and as such, should, IMHO, be able to go in a Flux repo 🙃

[monotek]

As it see seems to be a etcd limit, is this something that should be documented in the FAQ maybe or what would be your expectation?

Etcd's limit is 1 megabyte for applied resources but the annotation limit blocks this already at 262144 characters which are 0,25 megabyte.

From the k8s documentation (https://kubernetes.io/docs/concepts/configuration/configmap/#motivation):
"The data stored in a ConfigMap cannot exceed 1 MiB."

Therefore it would be nice to be able to use the actual etcd / configmap limit in fluxcd.
As mentioned earlier creating resources with "force replace" used when some annotation is set sounds like a proper solution.

For reference: #459

[MxNxPx]

@dholbach - appreciate you calling out the etcd limitation. As demonstrated in this repo: https://github.com/MxNxPx/flux2-issue, use of kubectl replace does allow this configmap to get created. So my expectation for an enhancement would be to expose an option that would specify the type of kubectl action for a given kustomization instead of ONLY doing an apply.

[palassis-8451]

@monotek seems to agree with the proposed approach... i am happy to test any proposed changes if the devs come up with a means to allow for a "kubectl replace" capability in Kustomizations. thanks!

[georg-pp]

The same issue when applying CRDs to cluster.
I'm trying to create CRDs with the manifest https://github.com/banzaicloud/koperator/releases/download/v0.18.0/kafka-operator.crds.yaml
I put it into flux gitops repo and was hit by:

flux-system	kustomization/kafka-operator-crds	False	apply failed: The CustomResourceDefinition "kafkaclusters.kafka.banzaicloud.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes	                                             	False 

I believe the same issue will arise with ArgoCD

[georg-pp]

@stefanprodan any suggestions?

[m1pl]

We're having the same Problem with this CRD. ArgoCD supports argocd.argoproj.io/sync-options: Replace=true. We tried spec.force in Flux but it didn't work. @stefanprodan Can we please have an option to recreate instead of apply?

[stefanprodan]

Starting with Flux 0.18 we've switch to Kubernetes server-side apply. This means Flux no longer uses kubectl, so the annotations limit is no longer an issue.

[m1pl]

Yep, works, thank you!

[LolloChicco]

Hi @stefanprodan and @m1pl , I'm using Flux 0.25.3 , and face with the same problem about annotations limit. Flux seems to not perform apply --server-side but only a classic apply. Any suggestion ? Thanks

[kingdonb]

Flux uses server-side apply since 0.18.0, please double check the version of Flux – not only the CLI binary that you have installed in your path on your workstation, but also the version that runs in the cluster.

flux version should provide the details about what version of each controller is running in your cluster. If it is out of date, please see the guide to upgrading that is most relevant to how you started your Flux installation, (or see https://fluxcd.io/docs/installation/#bootstrap-upgrade)

[tspearconquest]

Hello, I think I need this for gitops with crossplane.

Crossplane is designed so that an external resource it manages does not get recreated when a change requires the external resource to be recreated. It will only allow for a delete of the resource if it is removed from kubernetes. When using Flux with Crossplane, if I change, for example, the vmSize parameter of a kubernetes cluster node pool, my only option right now for recreation is to comment out the manifest from kustomization.yaml and merge it to git, wait for flux to reconcile, then uncomment and merge again.

If I could specify recreate, then all resources would be sent to the api server as a replacement for the deployed resources, and the old resources would be deleted from the api server, which would allow crossplane to delete and recreate those managed resources.

This is a supported feature in ArgoCD, apparently, so I'm basing the above on theory as I'm not an ArgoCD user.