Merge pull request #148 from fluxcd/monitoring-stack

Add monitoring stack and dashboards
fluxcd · Aug 5, 2020 · 0e80cd5 · 0e80cd5
2 parents b979e31 + 35a2099
commit 0e80cd5
Show file tree

Hide file tree

Showing 16 changed files with 1,734 additions and 0 deletions.
diff --git a/docs/_files/cp-dashboard-p1.png b/docs/_files/cp-dashboard-p1.png
diff --git a/docs/_files/cp-dashboard-p2.png b/docs/_files/cp-dashboard-p2.png
diff --git a/docs/guides/installation.md b/docs/guides/installation.md
@@ -328,3 +328,30 @@ tk create helmrelease sealed-secrets \
 --chart-name=sealed-secrets \
 --chart-version="^1.10.0"
 ```
+
+### Monitoring with Prometheus and Grafana
+
+The GitOps Toolkit comes with an optional monitoring stack.
+You can install the stack in the `gitops-system` namespace with:
+
+```yaml
+kustomize build github.com/fluxcd/toolkit/manifests/monitoring?ref=master | kubectl apply -f-
+```
+
+The monitoring stack is composed of:
+
+* Prometheus server - collects metrics from the toolkit controllers and stores them for 2h
+* Grafana dashboards - displays the control plane resource usage and reconciliation stats
+
+![](../_files/cp-dashboard-p1.png)
+
+![](../_files/cp-dashboard-p2.png)
+
+If you wish to use your own Prometheus and Grafana instances, then you can import the dashboards from
+[GitHub](https://github.com/fluxcd/toolkit/tree/master/manifests/monitoring/grafana/dashboards).
+
+!!! hint
+    Note that the toolkit controllers expose the `/metrics` endpoint on port `8080`.
+    When using Prometheus Operator you should create `PodMonitor` objects to configure scraping.
+    When Prometheus is running outside of the `gitops-system` namespace, you have to create a network policy
+    that allows traffic on port `8080` from the namespace where Prometheus is deployed.