"flux create source helm" randomly fails reconciliation with "no artifact was found"

[artem-nefedov]

Describe the bug

flux create source helm command would occasionally fail with an error:

◎ waiting for HelmRepository source reconciliation
✔ HelmRepository source reconciliation completed
✗ HelmRepository source reconciliation completed but no artifact was found

The actual HelmRepository object is fine and usable. If you immediately call flux reconcile source helm on it after encountering that error, it will succeed.

Steps to reproduce

1. Run flux create source helm
2. If bug isn't reproduced, delete HelmRelease and repeat step 1

Expected behavior

If there's a race condition, flux cli must properly wait for artifact to appear.

Screenshots and recordings

No response

OS / Distro

Linux

Flux version

v0.30.2

Flux check

fluxcd/helm-controller:v0.21.0
fluxcd/source-controller:v0.24.4

Git provider

No response

Container Registry provider

No response

Additional context

We've never seen this problem on Flux v0.28.5. After upgrading to Flux v0.30.2, we now see it regularly.

Code of Conduct

• I agree to follow this project's Code of Conduct

[somtochiama]

Hey @artem-nefedov ,
I have been unable to reproduce this. Can you give the helm repository you used if it is public?

[artem-nefedov]

Hi @somtochiama
Unfortunately, this is reproduced on a private repo.
Also, when I intentionally try to reproduce it (with either private or public repo), I don't have any success.
It does occasionally happen during our actual workflow, where we create the cluster, install flux, and create helm repo object, and I can't tell what makes the difference compared to isolated reproduction attempt.

You can either tell me what debug info I can collect, or leave/close the ticket until someone else has the same problem.

[somtochiama]

Can you see any logs from the source-controller? kubectl logs deploy/helm-controller and get the status of the helm repository kubectl get helmrepository <name> -oyaml

[somtochiama]

Looking at https://github.com/fluxcd/flux2/blob/main/cmd/flux/create_source_helm.go#L199, it seems there was really no artifact in helm repository. Maybe the helm chart isn't ready yet and some seconds later (when you run the second command) it is.

[artem-nefedov]

Looking at https://github.com/fluxcd/flux2/blob/main/cmd/flux/create_source_helm.go#L199, it seems there was really no artifact in helm repository. Maybe the helm chart isn't ready yet and some seconds later (when you run the second command) it is.

That could be the case, but then it begs the question: why can source-controller report that reconcile has finished before artifact is ready? It looks like there's a race condition in play here. Or, if that's a normal behaviour for the controller, why doesn't flux cli wait for artifact to appear?

For now, I've added a workaround to immediately call flux reconcile if flux create fails, e.g.:

flux create source helm ... || flux reconcile source helm ...

And, from what I see, the reconcile always succeeds.

[stefanprodan]

This should've been fixed in Flux v2.2