Add reconcile strategy for HelmCharts #308

arbourd · 2021-03-13T14:51:50Z

Adds ReconcileStrategy option allowing the HelmChart CRD to use the source's revision (ex: 0.1.0+a96eff73a282d408a960086d978ca97a0a1fb2ef) in place of the Chart version (ex: 0.1.0), by using the Revision option. ReconcileStrategy defaults to ChartVersion.

This is only applicable to GitRepository and Bucket sources as HelmRepository has a clearly defined contract that Charts are (should be) immutable.

This change would allow Flux users to determine if they want to install/upgrade on each change to the source. A more detailed workflow and reasoning as to why is below:

Use Case

We manage a single repository which houses a single chart that manages all microservices. Beside the ./templates lives values-<environment> for staging, test, dev, etc. Some developers keep their own values-name.yaml lying around elsewhere. We utilize Helm to its templating-fullest for production workloads and local develop (like bringing up databases in containers).

We chose not to package the Helm chart and ship it to a Helm repository. This would serve no purpose and just requires another step. We also chose to place the values files beside the values.yaml similar to how Bitnami once did. Currently, we use ArgoCD which uses a similar-to GitRepository source, templates on each change, and deploys when differences occur.

I feel like our use of Helm is optimal for GitOps:

We store the base (templates, values.yaml) in Git.
We store the overrides or patches beside the base (values-production.yaml) in the same Git repository.
We run CI on the base and overrides ensuring that the generated manifests comply with the Kubernetes API (against a KIND cluster).
We run CI to diff the local changed state against the remote state of the desired deployment target on pull-requests (similar to a terraform plan)
We update the chart many times a day mostly patching the values-env.yaml for changes to image tags.
By not versioning the Chart.yaml, changes to the base templates affect all environments immediately, meaning that all base changes must be backwards compatible.
We don't need to store the desired state (values) of our deployment in ConfigMaps, Secrets, Flux CRDs, ArgoCD CRDs, etc. It exists in a single place controlled, managed and audited by Git.

Without this change, Flux requires us to move values out of the Git repository and into other objects (mentioned above) or version the Chart.yaml with every change.

What happens with this change:

Set ReconcileStrategy to Revision
GitRepository changes reconcile with a new revision 0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc
Triggers a HelmChart reconciliation with the revision 0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc
Triggers a HelmRelease reconciliation with revision 0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc

This has been tested in an environment with a patched helm-controller. Changes to the HelmRelease and HelmChart CRD look like:

spec:
  chart:
    spec:
      chart: chart
      ignoreVersion: true
      sourceRef:
        kind: GitRepository
        name: flux-system
        namespace: flux-system
      valuesFile: chart/values-production.yaml
      version: '*'
  interval: 30s
  releaseName: prod
status:
  conditions:
  - lastTransitionTime: "2021-03-13T00:50:11Z"
    message: Release reconciliation succeeded
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-03-13T00:50:11Z"
    message: Helm upgrade succeeded
    reason: UpgradeSucceeded
    status: "True"
    type: Released
  helmChart: flux-system/default-example
  lastAppliedRevision: 0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc
  lastAttemptedRevision: 0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc
  lastAttemptedValuesChecksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
  lastReleaseRevision: 4
  observedGeneration: 1

api/v1beta1/helmchart_types.go

controllers/helmchart_controller.go

hiddeco

Helm has a strict SemVer requirement for charts since v3.5.2.

Instead of using the revision of the source, we should probably create a valid SemVer using the allowed build metadata, e.g v0.1.0-alpha+<commit SHA>. Using the full revision is not possible as:

Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-].
– SemVer spec

This change also requires a documentation update, as the migration guide currently states that this Helm Operator-like feature is no longer supported.

arbourd · 2021-03-13T18:19:20Z

Instead of using the revision of the source

Have I misunderstood how revision works? I'm not attempting to change the HelmChart CRD's Chart.Spec.Version, simply the revision metadata for tracking the differences between HelmChart CRDs. In my example, the Chart.Spec.Version is still 0.1.0 and never changes, satisfying Helm 3.5.2. My change versions the artifact differently and does not interact with the version of the chart at all.

The tarball from a HelmChart with revision: main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc yields this Chart.yaml:

apiVersion: v2
dependencies:
- condition: postgresql.enabled
  name: postgresql
  repository: https://charts.bitnami.com/bitnami
  version: 10.3.13
description: A Helm chart for an app
home: https://github.com/example/charts/tree/main/my-app
name: my-app
type: application
version: 0.1.0

The release looks like:

NAME      		NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART     	APP VERSION
release-name	default  	4       	2021-03-13 00:50:05.404387943 +0000 UTC	deployed	my-app-0.1.0

This change also requires a documentation update

Willing to do whatever is necessary if you're willing to accept the feature. 😁

hiddeco · 2021-03-18T20:31:50Z

You are correct, but I think it would be better to change the SemVer to include the build number than to copy the revision from the GitRepository. Reason being that it would not require any changes on the helm-controller side, and would be reflected in the Helm state as-well (as the chart version is persisted to the Helm storage).

kingdonb · 2021-04-07T18:17:01Z

I am still excited about this contribution, I think the compromise sounds fairly well fleshed out.

This will enable many uses of Helm Controller that have been problematic since changing from Helm Operator.

controllers/helmchart_controller.go

Legion2 · 2021-04-12T11:57:22Z

Would this also support the "ignore version" feature for local chart dependencies? For example, we maintain multiple helm charts in a git repository and they depend on each other using local file references. If we change the values.yaml of a dependency without updating the Chart.yaml currently the changes are not applied to the cluster.

kadzielawa · 2021-04-12T12:00:47Z

does it will work also for fluxcd in version 2 ?

hiddeco · 2021-04-12T12:33:30Z

does it will work also for fluxcd in version 2 ?

All work done in fluxcd/*-controller repositories is for Flux v2 / GitOps Toolkit components.

Would this also support the "ignore version" feature for local chart dependencies?

If they live in the same repository, then yes.

controllers/helmchart_controller.go

arbourd · 2021-04-19T18:14:26Z

Tests are failing so I think this has stopped working and I'll invest a bit more time on it this week.

stealthybox · 2021-04-22T19:30:02Z

After we merge this, we should update the modified lines from fluxcd/flux2#1067

hiddeco · 2021-05-19T11:04:42Z

@arbourd are you still committed to finishing this PR?

arbourd · 2021-05-19T15:34:25Z

@hiddeco Yup, just a little burnt out and taking a break.

arbourd · 2021-06-10T12:23:30Z

Hey @hiddeco I will get back to this, this weekend or next week. Thank you for your patience everyone.

hiddeco · 2021-06-10T13:49:54Z

Awesome, and take your time! I have been chewing myself on things like #361 for ages :-)

arbourd · 2021-08-04T21:39:25Z

Latest changes:

IgnoreVersion renamed to ReconcileStrategy
There are two strategies: ChartVersion and CommitSha. CommitSha is only functional with Git/Bucket based sources.
- CommitSha is nice and descriptive, but this also should work for Bucket based revisions as well which are technically not commits.
The Helm version is ammended (when CommitSha strategy is used) to add the metadata to the version. For example, 0.2.0 becomes 0.2.0+44b98beaeced64a1ea2d4bc704d884c2498b27e7
- I opted against a prerelease of alpha like the example above, so that Flux users can set their own prereleases without us impacting them

hiddeco · 2021-08-04T22:37:03Z

I will do a proper review of changes later, but would it be an idea to rename CommitSha to Revision so that it matches the terminology of the Artifact object?

arbourd · 2021-08-04T22:38:41Z

Yup! That shoulda been obvious 😅

api/v1beta1/helmchart_types.go

arbourd · 2021-08-06T19:27:15Z

Got this running in our staging environment nicely 😁

jack-evans · 2021-09-10T12:03:43Z

Is there plans to get this merged in soon? Really keen to use this feature and it's one of the dependencies my team needs to move to flux2

api/v1beta1/helmchart_types.go

controllers/helmchart_controller.go

hiddeco · 2021-09-10T13:13:53Z

@jack-evans originally the plan was to get this merged after some refactor work (see reconcilers-dev branch), but some unexpected events happened that caused distractions for some time.

@arbourd if you take my review comments into consideration, update the documentation in the docs/folder, and smash (and sign-off) your commits. I'd be happy to get this landed. Please note that the libgit2 dependencies are currently in a less-optimal shape, and your build may fail, I am addressing this at the moment.

arbourd · 2021-09-13T18:19:00Z

Anything I'm missing from a docs point of view @hiddeco? I've documented the values (consts) similar to other examples, and I have a draft in the real docs which will be more structured.

hiddeco · 2021-10-01T17:02:46Z

Before the weekend update after a long period of waiting.

While I was debugging the issue that suddenly popped up in the CI, it became clear that we needed a more permanent, safe and controllable way of building against libgit2. Getting a proper solution for this took awhile, battling the various issues along the way, but I expect #437 to unblock everything next week.

Thanks for your patience 🌻

stefanprodan

LGTM

Thanks @arbourd and @hiddeco

PS. I left a comment that's not blocking the merge.

stefanprodan · 2021-10-08T09:01:34Z

controllers/helmchart_controller.go

+
+	version := v.String()
+	if chart.Spec.ReconcileStrategy == sourcev1.ReconcileStrategyRevision {
+		// Isolate the commit SHA from GitRepository type artifacts by removing the branch/ prefix.


We need to extract this in fluxcd/pkg as it's being used in notification-controller also.

This commit adds a `ReconcileStrategy` field to the `HelmChart` resource, which allows defining when a new chart should be packaged and/or published if it originates from a `Bucket` or `GitRepository` resource. The two available strategies are: - `ChartVersion`: creates a new artifact when the version of the Helm chart as defined in the `Chart.yaml` from the Source is different from the current version. - `Revision`: creates a new artifact when the revision of the Source is different from the current revision. For the `Revision` strategy, the (checksum part of the) revision of the artifact the chart originatesfrom is added as SemVer metadata. A chart from a `GitRepository` with Artifact revision `main/f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`. A chart from a `Bucket` with Artifact revision `f0faacd5164a875ebdbd9e3fab778f49c5aadbbc` and a chart with e.g. SemVer `0.1.0` will be published as `0.1.0+f0faacd5164a875ebdbd9e3fab778f49c5aadbbc`. Signed-off-by: Dylan Arbour <arbourd@users.noreply.github.com>

hiddeco

Made some tiny changes to the doc blocks and commit message, but the code was all looking OK @arbourd. Thank you very much 🙇

kadzielawa · 2021-10-08T09:33:00Z

good job guys ! @hiddeco @stefanprodan

arbourd commented Mar 13, 2021

View reviewed changes

api/v1beta1/helmchart_types.go Outdated Show resolved Hide resolved

arbourd commented Mar 13, 2021

View reviewed changes

controllers/helmchart_controller.go Outdated Show resolved Hide resolved

hiddeco reviewed Mar 13, 2021

View reviewed changes

arbourd commented Apr 10, 2021

View reviewed changes

controllers/helmchart_controller.go Outdated Show resolved Hide resolved

arbourd commented Apr 19, 2021

View reviewed changes

controllers/helmchart_controller.go Outdated Show resolved Hide resolved

arbourd commented Apr 19, 2021

View reviewed changes

controllers/helmchart_controller.go Outdated Show resolved Hide resolved

hiddeco mentioned this pull request May 19, 2021

HelmChart is not reconciled when updating GitRepository fluxcd/flux2#1427

Closed

arbourd changed the title ~~Allow HelmChart to use Source revision over version~~ Add Reconcile Strategy Aug 4, 2021

arbourd marked this pull request as ready for review August 4, 2021 21:40

arbourd commented Aug 5, 2021

View reviewed changes

api/v1beta1/helmchart_types.go Outdated Show resolved Hide resolved

hiddeco reviewed Sep 10, 2021

View reviewed changes

api/v1beta1/helmchart_types.go Outdated Show resolved Hide resolved

controllers/helmchart_controller.go Outdated Show resolved Hide resolved

hiddeco added the area/git Git related issues and pull requests label Sep 10, 2021

hiddeco added area/helm Helm related issues and pull requests enhancement New feature or request labels Sep 10, 2021

hiddeco changed the title ~~Add Reconcile Strategy~~ Add reconcile strategy for HelmCharts Sep 10, 2021

arbourd requested a review from hiddeco September 11, 2021 19:56

arbourd mentioned this pull request Sep 11, 2021

Add reconcile strategy documentation fluxcd/website#516

Merged

stefanprodan approved these changes Oct 8, 2021

View reviewed changes

hiddeco approved these changes Oct 8, 2021

View reviewed changes

hiddeco merged commit d382eca into fluxcd:main Oct 8, 2021

arbourd deleted the ignore-version branch October 8, 2021 14:57

hiddeco mentioned this pull request Oct 8, 2021

Document ReconcileStrategy field for HelmRelease fluxcd/flux2#1911

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add reconcile strategy for HelmCharts #308

Add reconcile strategy for HelmCharts #308

arbourd commented Mar 13, 2021 •

edited

Loading

hiddeco left a comment •

edited

Loading

arbourd commented Mar 13, 2021 •

edited

Loading

hiddeco commented Mar 18, 2021

kingdonb commented Apr 7, 2021

Legion2 commented Apr 12, 2021

kadzielawa commented Apr 12, 2021 •

edited

Loading

hiddeco commented Apr 12, 2021

arbourd commented Apr 19, 2021

stealthybox commented Apr 22, 2021

hiddeco commented May 19, 2021

arbourd commented May 19, 2021

arbourd commented Jun 10, 2021

hiddeco commented Jun 10, 2021

arbourd commented Aug 4, 2021 •

edited

Loading

hiddeco commented Aug 4, 2021

arbourd commented Aug 4, 2021

arbourd commented Aug 6, 2021

jack-evans commented Sep 10, 2021

hiddeco commented Sep 10, 2021

arbourd commented Sep 13, 2021

hiddeco commented Oct 1, 2021 •

edited

Loading

stefanprodan left a comment

stefanprodan Oct 8, 2021

hiddeco left a comment

kadzielawa commented Oct 8, 2021

Add reconcile strategy for HelmCharts #308

Add reconcile strategy for HelmCharts #308

Conversation

arbourd commented Mar 13, 2021 • edited Loading

hiddeco left a comment • edited Loading

Choose a reason for hiding this comment

arbourd commented Mar 13, 2021 • edited Loading

hiddeco commented Mar 18, 2021

kingdonb commented Apr 7, 2021

Legion2 commented Apr 12, 2021

kadzielawa commented Apr 12, 2021 • edited Loading

hiddeco commented Apr 12, 2021

arbourd commented Apr 19, 2021

stealthybox commented Apr 22, 2021

hiddeco commented May 19, 2021

arbourd commented May 19, 2021

arbourd commented Jun 10, 2021

hiddeco commented Jun 10, 2021

arbourd commented Aug 4, 2021 • edited Loading

hiddeco commented Aug 4, 2021

arbourd commented Aug 4, 2021

arbourd commented Aug 6, 2021

jack-evans commented Sep 10, 2021

hiddeco commented Sep 10, 2021

arbourd commented Sep 13, 2021

hiddeco commented Oct 1, 2021 • edited Loading

stefanprodan left a comment

Choose a reason for hiding this comment

stefanprodan Oct 8, 2021

Choose a reason for hiding this comment

hiddeco left a comment

Choose a reason for hiding this comment

kadzielawa commented Oct 8, 2021

arbourd commented Mar 13, 2021 •

edited

Loading

hiddeco left a comment •

edited

Loading

arbourd commented Mar 13, 2021 •

edited

Loading

kadzielawa commented Apr 12, 2021 •

edited

Loading

arbourd commented Aug 4, 2021 •

edited

Loading

hiddeco commented Oct 1, 2021 •

edited

Loading