Query-frontend: Add experimental query blocker #5609
Conversation
wilfriedroset
force-pushed
the
queries-blocker
branch
3 times, most recently
from
e0321fb
to
a086fbd
Compare
wilfriedroset
force-pushed
the
queries-blocker
branch
3 times, most recently
from
2f81022
to
cde3fe8
Compare
|
|
Sorry about closing the PR; I have no idea what keystroke I mistakenly just used. Eek. |
wilfriedroset
force-pushed
the
queries-blocker
branch
from
cde3fe8
to
ca07837
Compare
|
thank you @osg-grafana, I've taken into account your remarks. |
| # Configure queries to block | ||
|
|
||
| In certain situations, you might want to control what queries are being sent to your Mimir installation. These queries | ||
| might be intentionally or unintentionally expensive to run, and they might affect the overall stability or cost of running |
There was a problem hiding this comment.
Hi @wilfriedroset, perhaps for my personal learning, but why would a query ever be intentionally expensive to run? It sounds like there is something underneath this idea and the result is that a given query is expensive to run.
There was a problem hiding this comment.
From a service provider, we might be exposed to denial of service, this is where such features are useful.
For example, a single user looking to cause harm to your service could storm the Mimir cluster with queries that would not be stopped by the current limits.
In that event, either the cluster is scaled or the quality of service decrease.
Most of this work is inspired and/or copied from Loki.
Loki's documentation contains the same phrase: https://grafana.com/docs/loki/latest/operations/blocking-queries/
wilfriedroset
force-pushed
the
queries-blocker
branch
from
ca07837
to
7f41cc7
Compare
|
|
||
| for _, block := range blocks { | ||
| if strings.TrimSpace(block.Pattern) == strings.TrimSpace(query) { | ||
| level.Warn(logger).Log("msg", "query blocker matched with exact match policy", "query", query) |
There was a problem hiding this comment.
I'd make this an Info log line. Everything is working as intended.
There was a problem hiding this comment.
On another note: it might be useful to identify which rule is blocking the query. Maybe something as simple as the index of the rule will be enough to debug block configurations.
|
The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase |
wilfriedroset
force-pushed
the
queries-blocker
branch
4 times, most recently
from
c3e4e26
to
a70f9b1
Compare
|
Sorry about the delay @dimitarvdimitrov I was on PTO. |
wilfriedroset
force-pushed
the
queries-blocker
branch
from
a70f9b1
to
ecae801
Compare
wilfriedroset
force-pushed
the
queries-blocker
branch
3 times, most recently
from
7f8d3ba
to
d64df6b
Compare
This commit implements the following proposal: grafana#5029 Signed-off-by: Wilfried Roset <wilfriedroset@users.noreply.github.com>
wilfriedroset
force-pushed
the
queries-blocker
branch
from
d64df6b
to
d7425bf
Compare
There was a problem hiding this comment.
LGTM, great work! And thanks for addressing my feedback.
@osg-grafana do you have any other remarks on the docs or are we good to merge this?
There was a problem hiding this comment.
Thanks @wilfriedroset ! Do you mind adding the feature to the list of experimental features listed at docs/sources/mimir/configure/about-versioning.md?
What this PR does
Add experimental query blocker
Which issue(s) this PR fixes or relates to
This commit implements the following proposal: #5029
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]