-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
query-frontend: stop using creation_grace_period to limit query length #8075
query-frontend: stop using creation_grace_period to limit query length #8075
Conversation
3144's reason for truncating queries is to not fill the scheduler queue with effectively useless queries. Queries can legitimately be querying into the future without there being samples ingested for that time. For example `predict_linear` used on a longer term scale. Since the impact is per-tenant I think undoing this clamping is reasonable - it will only impact the tenant sending the large queries. Perhaps worth noting that `-query-frontend.max-total-query-length` still applies. Reviewing 7496 made me realize this. Fixes 5445 Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
I'm not convinced removing it, sorry. IIRC, this protection was introduced after an incident and now we're removing that protection. Maybe we can convert it into a new dedicated limit, which is more generous (e.g. 1 day ahead is fine, 100 years ahead is not). |
we still have
I couldn't find the problem this caused. I think you just noticed it happening, but the actual problem was related to mmap and lazy loading. |
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @dimitarvdimitrov's arguments: querying into the future should be cheaper than querying into the past, and still limited by the max query length.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, your arguments make sense.
What this PR does
Reviewing #7496 made me realize this.
#3144's reason for truncating queries is to not fill the scheduler queue with effectively useless queries. Some queries can legitimately be querying into the future without there being samples ingested for that time. For example
predict_linear
used on a longer term scale.Since the impact of undoing this change in this PR is per-tenant I think it is reasonable - it will only impact the tenant sending the large queries. Perhaps worth noting that
-query-frontend.max-total-query-length
still applies.Which issue(s) this PR fixes or relates to
Fixes #5445
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]
.about-versioning.md
updated with experimental features.