FluxCD fails to create configmaps greater than 262144 characters

[monotek]

I use flux2 0.2.2.

I try to add a ConfigMap with a Grafana dashboard, but get the following error:

...The ConfigMap \"k8s-node-exporter\" is invalid: metadata.annotations: Too long: must have at most 262144 characters\n"}

This is not rely a fluxcd problem as "kubectl apply" also fails to create the configmap:

kubectl apply -f node-exporter-full.yaml 
The ConfigMap "k8s-node-exporter" is invalid: metadata.annotations: Too long: must have at most 262144 characters

What works is to add the configmap via "kubectl create" and updating it with "kubectl replace"

kubectl create -f node-exporter-full.yaml 
configmap/k8s-node-exporter created

kubectl replace -f node-exporter-full.yaml 
configmap/k8s-node-exporter replaced

So i guess it could work if fluxcd would support some annotation to choose the method for creating the file via "create" / "replace".

Would adding something like this work or did i miss any other available workaround?

For reference: fluxcd/flux#1962

[stefanprodan]

Duplicate of: fluxcd/kustomize-controller#122

[monotek]

Yesterday i've run in this again (with flux v0.12.3), trying to use kustomize configMapGenerator like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namePrefix: grafana-dashboard-
namespace: monitoring
generatorOptions:
  disableNameSuffixHash: true
  labels:
    grafana_dashboard: "yes"
configMapGenerator:
  - name: node-exporter
    files:
      - node-exporter-full_rev22.json

The nodeexporter dashboard can be found here: https://grafana.com/grafana/dashboards/1860

The newly introduced force flag (fluxcd/kustomize-controller#271) did not help, as i'm not trying to change an immutable field. So imho marking this ticket as a duplicate is not entirely correct.

Kubectl apply fails, also with force:

kustomize build . | kubectl apply --force -f -
The ConfigMap "grafana-dashboard-node-exporter" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Imho the only way to get this work is to be able to use something like the command below for initial creation and updates of the resource:

kustomize build . | kubectl replace --force -f -

Idealy i'd like to manage all resources with fluxcd.
So do you see any way we could achieve this?
How to you pass resources like dashboard configmaps to your clusters?
Using a separate cd pipeline for this feels wrong to me.

[stefanprodan]

I don't think etcd is the right place to store such large objects, place them in a container image and copy them to Grafana storage using an init container.

[monotek]

Etcd's limit is 1 megabyte for applied resources but the annotation limit blocks this already a lot earlier.

From the k8s documentation (https://kubernetes.io/docs/concepts/configuration/configmap/#motivation):
"The data stored in a ConfigMap cannot exceed 1 MiB."

Therefore it would be nice to be able to use the actual etcd / configmap limit.

I thought about the init container workaround to but it still feels a bit bad to leave the gitops pattern here.

[monotek]

Fore reference: #1382

[Frizlab]

@stefanprodan CRD also often fall into the category of big objects.