Cope when syncing a CRD and resources using the CRD

[drewz-aware]

This bug was in flux v1 and can be found here : fluxcd/flux#1825
This bug was fixed in v1 here : fluxcd/flux#1876

"If you add a CRD as well as resources that use the CRD, in one commit (or in commits since the last sync), fluxd will stop being able to sync.

What I think is happening is that the namespacer will attempt to look up the scope of the custom resources, and since the definition has not yet been created, it will fail. At present this will stop the sync, and fluxd will not be able to make progress."

With a fresh V2 install we seem to be running into this same bug.

[kingdonb]

Duplicate report, I think this has come up at least once before, here is a similar report: #523

While this was considered a bug in Flux v1 where there was no validation routine built in, and applies were allowed to be "eventually consistent," in Flux v2 this is not a bug as it is due to Flux's proper validating that we encourage users to enable.

Flux v2 with validation enabled will need to show that the entire commit is valid before it can apply any part of the commit to the cluster. This validation can be disabled with validate: false, but following the train of thought in the thread you posted:

I think this could be work-around by having a separate flux instance that managed special things, like CRDs and potentially other non-namespaces things, like actual Namespaces and ClusterRoles/ClusterRoleBindings

This was a kludgy work-around in Flux v1, and less than ideal, as there is a fair amount of overhead to running more than one single Flux instance. In Flux v2, which supports multiple syncs and multiple repos as first-class entities, that config structure is no longer a work-around, it's a straightforward expression of the big design idea.

https://fluxcd.io/docs/components/kustomize/kustomization/#kustomization-dependencies

Flux 2 users are expected to organize their repositories and sort resources into "infrastructure" and "tenants" which can be loaded respectively, earlier and later, to avoid this type of issue... Kustomize controller provides dependency ordering to ensure that the "depended-on" resources are sync'ed and potentially even health-checked ahead of time, somewhere before the downstream "tenant" or "dependent" resources start to get applied to the cluster.

I was looking for a good explanation of this in the FAQ and I will concede this could potentially receive better coverage in the docs. There aren't practically any mentions of CRDs in the migration guides also, for example.

The short version is, you now have to sync your CRDs before the Custom Resources they define, so that Flux v2 can help with validating those CRs before they are applied. If you have implemented validation already placed ahead of this in a CI pipeline and are using merge checks to prevent this type of failure, it can be more than reasonable to disable the validation in Flux.

[hiddeco]

In addition to @kingdonb's excellent response; a bit of friendly community gatekeeping, but can you please leave triaging of issues up to the maintainers?

As we have seen dozens of releases in the past months, issues with a high user impact are unlikely unless they are related to a newly introduced feature (which this is not).