Skip to content

Commit

Permalink
Fix GossipMembersMismatch misfiring with remote ruler eval (#2159)
Browse files Browse the repository at this point in the history
* Fix GossipMembersMismatch misfiring with remote ruler eval

The alert doesn't take into account the queries that now have ruler-
prepended to their name. This PR fixes it

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Simpler regexes

* Update computed mixin
  • Loading branch information
dimitarvdimitrov committed Jun 21, 2022
1 parent d58da4a commit 9f46b6f
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion operations/mimir-mixin-compiled/alerts.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ groups:
message: Mimir instance {{ $labels.pod }} in {{ $labels.cluster }}/{{ $labels.namespace
}} sees incorrect number of gossip members.
expr: |
avg by (cluster, namespace) (memberlist_client_cluster_members_count) != sum by (cluster, namespace) (up{job=~".+/(alertmanager|compactor|distributor|ingester.*|querier.*|ruler|store-gateway.*|cortex|mimir)"})
avg by (cluster, namespace) (memberlist_client_cluster_members_count) != sum by (cluster, namespace) (up{job=~".+/(alertmanager|compactor|distributor|ingester.*|querier.*|ruler|ruler-querier.*|store-gateway.*|cortex|mimir)"})
for: 15m
labels:
severity: warning
Expand Down
2 changes: 1 addition & 1 deletion operations/mimir-mixin/config.libsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
ruler_query_frontend: '(ruler-query-frontend.*)', // Match also custom ruler-query-frontend deployments.
query_scheduler: 'query-scheduler.*', // Not part of single-binary. Match also custom query-scheduler deployments.
ruler_query_scheduler: 'ruler-query-scheduler.*', // Not part of single-binary. Match also custom query-scheduler deployments.
ring_members: ['alertmanager', 'compactor', 'distributor', 'ingester.*', 'querier.*', 'ruler', 'store-gateway.*', 'cortex', 'mimir'],
ring_members: ['alertmanager', 'compactor', 'distributor', 'ingester.*', 'querier.*', 'ruler', 'ruler-querier.*', 'store-gateway.*', 'cortex', 'mimir'],
store_gateway: '(store-gateway.*|cortex|mimir)', // Match also per-zone store-gateway deployments.
gateway: '(gateway|cortex-gw|cortex-gw-internal)',
compactor: 'compactor.*|cortex|mimir', // Match also custom compactor deployments.
Expand Down

0 comments on commit 9f46b6f

Please sign in to comment.