Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mimir query engine: report queries rejected due to hitting the memory consumption limit in the cortex_querier_queries_rejected_total metric #8303

Merged
merged 4 commits into from
Jun 7, 2024

Conversation

charleskorn
Copy link
Contributor

@charleskorn charleskorn commented Jun 7, 2024

What this PR does

This PR builds on #8230, and adds incrementing the cortex_querier_queries_rejected_total{reason="max-estimated-memory-consumption-per-query"} metric when a query is rejected for exceeding the per-query memory consumption limit.

Which issue(s) this PR fixes or relates to

(none)

Checklist

  • Tests updated.
  • [n/a] Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • [n/a] about-versioning.md updated with experimental features.

@charleskorn charleskorn marked this pull request as ready for review June 7, 2024 05:51
@charleskorn charleskorn requested a review from a team as a code owner June 7, 2024 05:51
Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Minor nit: would be nice to see a test where 3 out of 4 queries fail and the counter is as we expect (make sure we don't double count them etc)

@charleskorn
Copy link
Contributor Author

lgtm. Minor nit: would be nice to see a test where 3 out of 4 queries fail and the counter is as we expect (make sure we don't double count them etc)

Good idea, added in a0235fe

@charleskorn charleskorn enabled auto-merge (squash) June 7, 2024 06:46
@charleskorn charleskorn merged commit 7adf4e1 into main Jun 7, 2024
29 checks passed
@charleskorn charleskorn deleted the charleskorn/emit-rejected-metric-for-memory-limit branch June 7, 2024 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants