Wait for CRDs existence at runtime

[nlewo]

kapp cannot deploy Gatekeeper with a Gatekeeper configuration because the configuration uses a CRD created by the Gatekeeper controller itself.

For instance, in https://github.com/open-policy-agent/gatekeeper#constraint-templates, a template is created. This template is used by the Gatekeeper controller to create the CRD K8sRequiredLabels. This kind can be used by the user to create constraints such as https://github.com/open-policy-agent/gatekeeper#constraints.

So, currently, we first need to deploy the Gatekeeper controller, and then add the configuration custom resource in the app. Otherwise, kapp refuses to deploy the app because the CRD K8sRequiredLabels is missing.

In this kind of scenario, I think kapp should be able to wait for CRDs at runtime. This could be specified as a annotation in the custom resource metadata.

WDYT?

[cppforlife]

hmm, i dont think ive seen a project that dynamically creates crd (for a good reason).

2 reasons currently kapp cares about crds:
a. should it insert namespace into CR (depending on if CRD is namespaced)
b. determine when it should create/delete CR (delete is an interesting case as well)

given kapp's separation of diff stage from apply, it's challenging to deal with point (a), given that info isnt available in the diffing stage, unless we ask users to specify an annotation that lets us know if it's namespaced or not (though this becomes duplicate data).

regarding point (b), creation could be dealt with as you've described (may be with addition of timeout). deletion is a bit harded since we dont know when CRD is deleted, so kapp cant help you to make sure CRs are deleted before CRD (may be k8s have some internal logic to deal with that gracefully).

i think your suggestion of adding annotations is definitely one way. another alternative that might be interesting is adding a "cluster-fulfilled" resource (in this example for a CRD). by having a dedicated resource we could plug it into order hierarchy (https://github.com/k14s/kapp/blob/develop/docs/apply-ordering.md). ill think more about this while i'm attending kubecon.

[voor]

Dex dynamically creates its CRDs as well.

[kferrone]

In my case the CRDs are created by FluxCDs Helm Operator. A ton of Helm charts will bundle their CRDs and therefore the CRDs are dynamically created when I use the HelmRelease kind from FluxCD. So kapp can't really see what is included in a HelmRelease and I can't tell a custom resource to simply wait for the operator from the helmRelease to complete because the CRDs are not in the cluster yet.

[pivotaljohn]

The Carvel team is taking a serious swing at addressing this need generically (i.e. as a first-class feature of the order hierarchy): #194

[praveenrewar]

We have added support of kapp.k14s.io/exists annotation (available in v0.43.0) which can be used to wait for resources that are not owned by kapp.
For the above mentioned example of gatekeeper, we might wanna add a few change-groups and change-rules as well (to deploy the Constraint Templates after the pods go into the ready state). Please refer to this gist for reference.

[renuy]

@nlewo, Do try this out, and let us know how it goes.
Thanks

[nlewo]

@renuy I'm sorry, i moved to some other stuffs and no longer use kapp (not because i find something better, just because i no longer have the need).

btw, thx for the ping ;)