-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
query: Staleness problem #2608
Comments
What is the ceph version? |
CEPH does not matter, because thanos-stores return all data to query, and only in prometheus library points marked as staleness. |
At a minimum, it would be good to add the |
I think that we need in dynamic lookback-delta inpdepended on resolution, forexample resolution/2. |
Well. The main problem is that we can use different resolution in single PromQL eval (: So it can be [1h of raw data, 2w of 1h resolution, and 5h of 5m resolution] combined. So I think we might need to think of something in the PromQL itself. @brian-brazil do you know how hard would be that? Also we can temporarily add lookback delta per query as well 🤔 |
Varying resolution within one query is unlikely to wrok. What I'd do is present that to PromQL that looks real from the downsampled data - e.g. here you might provide interpolated samples every 1m. It kinda depends on what the query is though. |
@bwplotka @brian-brazil |
Can you elaborate more @brian-brazil ? So essentially you would actually for each downsampled data, actually expand it to have samples every 1m, fake interval? 🤔 What would be the corner cases? Why it depends on query? Alternatively we could have 3 PromQL engines in Querier and chose what to use based on the returned data. Then we can evaluate for the given periods and contact the results. However for large steps and intervals, it would be most likely bad.... |
Yes, something like that.
For e.g. sum_over_time you need different data than count_over_time to produce the desired result. |
Looks like @IKSIN we could try that in |
Ok! I try do it ) |
I am pretty sure we need special iterator for downsampled chunks. |
This is already well handled. |
Hello 👋 Looks like there was no activity on this issue for last 30 days. |
Closing for now as promised, let us know if you need this to be reopened! 🤗 |
we still work on this, let's reopen |
Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com>
BTW do you know we can now configure stalenees Lookback delta? However we might want to adjust it for different resolutions indeed |
@bwplotka As I remember staleness lookback delta is not something new. Or it was changed recently somehow? |
We just allow users to configure it on Querier from flag that's it. |
Closes thanos-io#2608 This allows queries with large step to make use of downsampled data. Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com>
Well, here's my attempt at it: #3277 |
Closes thanos-io#2608 This allows queries with large step to make use of downsampled data. Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com>
* query: introduce dynamic lookback interval Closes #2608 This allows queries with large step to make use of downsampled data. Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Fix minor checks Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Append changelog Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Add missing copyright Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Use pre-defined downsampling resolution constatns Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Use dynamic lookback delta by default Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Rename defaultEngine to rawEngine Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Merge engineFunc and newEngine into single engineFactory Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Remove query.dynamic-lookback-delta from docs Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Rename test Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Review fixes Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
* query: introduce dynamic lookback interval Closes thanos-io#2608 This allows queries with large step to make use of downsampled data. Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Fix minor checks Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Append changelog Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Add missing copyright Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Use pre-defined downsampling resolution constatns Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Use dynamic lookback delta by default Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Rename defaultEngine to rawEngine Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Merge engineFunc and newEngine into single engineFactory Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Remove query.dynamic-lookback-delta from docs Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Rename test Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Review fixes Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Oghenebrume50 <raphlbrume@gmail.com>
Thanos, Prometheus and Golang version used:
thanos: v0.12.0
Object Storage Provider:
private CEPH (S3)
What happened:
See on
end_input
time andresolution
:Staleness functionality in prometheus library get rid of some points returned from thanos-stores.
What you expected to happen:
Return all data from store on any time_range
How to reproduce it (as minimally and precisely as possible):
see on screenshots.
Full logs to relevant components:
Anything else we need to know:
I think, that we have few ways to resolve problem:
LookbackDelta
parameter > 5 min (need check)The text was updated successfully, but these errors were encountered: