Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete stale metrics on object delete #1183

Merged
merged 1 commit into from
Aug 14, 2023
Merged

Conversation

darkowlzz
Copy link
Contributor

@darkowlzz darkowlzz commented Jul 31, 2023

Depends on fluxcd/pkg#612 .

Move record suspend metrics next to readiness and duration metrics so
that it gets recorded along with others metrics always at the end and the
metrics delete, which requires the knowledge of deleted finalizers,
applies to suspend metrics too.

HelmRepository cache event metrics for a given helmrepo also continues
to be exported even after the object is deleted. This change deletes
the cache event metrics when the object is deleted.

Before this change, the following sample metrics continued to be exported even after the associated object is deleted:

# HELP gotk_reconcile_condition The current condition status of a GitOps Toolkit resource reconciliation.
# TYPE gotk_reconcile_condition gauge
gotk_reconcile_condition{kind="GitRepository",name="flux-monitoring",namespace="flux-system",status="False",type="Ready"} 0
gotk_reconcile_condition{kind="GitRepository",name="flux-monitoring",namespace="flux-system",status="True",type="Ready"} 1
gotk_reconcile_condition{kind="GitRepository",name="flux-monitoring",namespace="flux-system",status="Unknown",type="Ready"} 0
gotk_reconcile_condition{kind="HelmChart",name="monitoring-kube-prometheus-stack",namespace="monitoring",status="False",type="Ready"} 0
gotk_reconcile_condition{kind="HelmChart",name="monitoring-kube-prometheus-stack",namespace="monitoring",status="True",type="Ready"} 1
gotk_reconcile_condition{kind="HelmChart",name="monitoring-kube-prometheus-stack",namespace="monitoring",status="Unknown",type="Ready"} 0
...
# HELP gotk_suspend_status The current suspend status of a GitOps Toolkit resource.
# TYPE gotk_suspend_status gauge
gotk_suspend_status{kind="GitRepository",name="flux-monitoring",namespace="flux-system"} 0
gotk_suspend_status{kind="HelmChart",name="monitoring-kube-prometheus-stack",namespace="monitoring"} 0
...
# HELP gotk_cache_events_total Total number of cache retrieval events for a Gitops Toolkit resource reconciliation.
# TYPE gotk_cache_events_total counter
gotk_cache_events_total{event_type="cache_hit",name="podinfo",namespace="default"} 3
gotk_cache_events_total{event_type="cache_miss",name="podinfo",namespace="default"} 1
...

With this change, they get deleted once the associated object is deleted.

@darkowlzz darkowlzz added the area/helm Helm related issues and pull requests label Jul 31, 2023
@darkowlzz darkowlzz force-pushed the stale-helm-cache-metrics branch 2 times, most recently from f5a4b54 to d80805f Compare July 31, 2023 21:57
@darkowlzz darkowlzz changed the title helmrepo: Delete cache metrics on object delete Delete stale metrics on object delete Aug 9, 2023
@darkowlzz darkowlzz force-pushed the stale-helm-cache-metrics branch 3 times, most recently from 46536da to 0734b61 Compare August 10, 2023 14:00
@darkowlzz darkowlzz marked this pull request as ready for review August 10, 2023 14:01
@darkowlzz darkowlzz added enhancement New feature or request and removed area/helm Helm related issues and pull requests labels Aug 10, 2023
@darkowlzz darkowlzz force-pushed the stale-helm-cache-metrics branch 5 times, most recently from 5ac82b0 to 97cbbe0 Compare August 11, 2023 13:56
Move record suspend metrics next to readiness and duration metrics so
that it gets recorded along with others always at the end and the
metrics delete, which requires the knowledge of deleted finalizers,
applies to suspend too.

HelmRepository cache event metrics for a given helmrepo also continues
to be exported even after the object is deleted. This change deletes
the cache event metrics when the object is deleted.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
@darkowlzz darkowlzz merged commit 36ca723 into main Aug 14, 2023
10 checks passed
@darkowlzz darkowlzz deleted the stale-helm-cache-metrics branch August 14, 2023 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants