Upgrading to v2.0.1 safely

[robertpcontreras-ts]

Hello,

We are currently working on upgrading out fluxcd versions to v2.0.1 from v0.37.0.

We have 2 development clusters and this process went smoothly on one of the clusters but on the other cluster, all of our workloads were deleted during the upgrade.

We have typically upgraded our clusters using the following command:

flux install \                   
  --components-extra=image-reflector-controller,image-automation-controller \
  --export > ./clusters/dev/flux-system/flux-components.yaml

We then commit and push these updated manifests to the repo and allow flux to update itself.

However, having migrated to GitHub recently we wanted to migrate to the flux bootstrap github command. So we used the following steps to try and achieve this.

brew upgrade fluxcd/tap/flux

flux uninstall --namespace=flux-system

export GITHUB_TOKEN=NEW_GITHUB_TOKEN

# backup the existing folder
mv ./clusters/pre-prod/flux-system ./clusters/_old_pre-prod/flux_system

git add -A && git commit -m "moving the old pre-prod cluster flux system folder as this is not compatible with the new one" && git push

# re install flux using the github bootstrap...
flux bootstrap github \
  --owner=xxx \
  --repository=fleet \
  --path=clusters/pre-prod \
  --branch=pre-prod \
  --components-extra=image-reflector-controller,image-automation-controller

As mentioned above, this seemed to work without issue on our dev cluster, but after running this process on our pre prod cluster, all resources built by kustomization files in clusters/pre-prod where deleted.

Thanks to Flux and GitOps we were able to recover this pretty quickly but this is obviously something that we cannot risk when we upgrade our production cluster.

Please can you advise how we should approach this?

Thankfully, we do have other non prod clusters to further test approaches on so would appreciate any suggestions.

Kind regards

[kingdonb]

If you still have the commit that deleted everything in pre-prod, it should be pretty trivial to inspect it with flux build or flux diff and figure out what happened here.

mv ./clusters/pre-prod/flux-system ./clusters/_old_pre-prod/flux_system

git add -A && git commit -m "moving the old pre-prod cluster flux system folder as this is not compatible with the new one" && git push

I'd start looking here though. The commands above definitely have potential to delete your whole cluster, this isn't a procedure I would have advised you on in this way. If you had prune enabled on the flux-system kustomization, and if there were any files left in this directory at all, or perhaps an intermediary state was reconciled that left this directory empty, then I would expect this very likely deletes your cluster and all its content.

This bit of trivia might explain what was the difference: in a Flux Kustomization, if your path: ./clusters/pre-prod is present but empty, this is treated differently than if the path is missing.

When the Flux kustomize controller finds you pointed it at a path and the path is missing, it assumes this is a mistake. It won't just start deleting everything. But if you point it at an empty directory, it doesn't assume it's a mistake. It will go ahead and delete everything.

I would strongly advise putting pre-prod back the way it was when you started, then role-playing and running back through exactly what steps you did, until you understand just how this happens (in order to ensure you can avoid the same problem in production!)

See this guide also, from the FAQ on safely moving resources:

https://fluxcd.io/flux/faq/#how-can-i-safely-move-resources-from-one-dir-to-another

In case you know why you need to move those resources around, and you don't want to delve into the reasons why it's needed, it could simply be that "prune" is enabled in your pre-prod environment and it's not enabled in the dev environment.

I would recommend having prune enabled on all environments.

[robertpcontreras-ts]

@kingdonb thanks for the reply. really appreciate it.

I understand what your saying and I'd fully expect this behaviour apart from that fact that flux uninstall --namespace=flux-system was ran before the step you are referring to above.

Therefore flux would not have been installed at this point to pull that update and apply changes to the cluster.

Regarding prune... if we have this set to false on all our kustomizations (just for the purposes of this migration) will that ensure that flux does not attempt to delete any resources? therefore making this a safer operation?

[kingdonb]

If Flux was not installed then how did flux delete the cluster? You're saying that on the next bootstrap, flux started deleting before it did anything else? That's very strange and I want to see it reproduced.

Yes, setting prune to false is one way. Suspending the resources should be another way. But in the end you don't want to use these hacks to break the resource accounting since suspending will only temporarily avoid the issue, reconciling will happen eventually. And disabling prune has the potential to leave orphaned resources behind. I would really like to know how this can happen. My guess is that the flux system namespace got deleted while Flux was still running, but it's just a guess without a repro.

Here's one way it could happen: when you ran flux uninstall the Namespace went "Terminating" – then you installed Flux again, but the namespace was still set to terminating. So the resources that Flux created (with prune enabled) got terminated and all the controllers enacted the garbage collection. Maybe if you can reproduce this condition, we should add a check to flux bootstrap that insists on waiting until the namespace stops Terminating for sanity checking. It is possible you discovered a weird edge case here.

[kingdonb]

You mentioned flux uninstall --namespace=... -- another important question maybe, are you running multiple instances of Flux on the cluster?

[robertpcontreras-ts]

Thanks again for the reply.

I made sure that the namespace was fully terminated before continuing (we have a script to force termination).

And nope, we don't have multiple versions of flux running in the cluster.

I am going to look into setting prune to false for our remaining tests. I agree this isn't a long term solution but we will set this back to true once we have upgraded the versions and migrated to using flux bootstrap github going forward.