-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ingester: implement request size instance limit #6492
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice job! LGTM (modulo a nit)
if err := i.checkAvailable(); err != nil { | ||
return err | ||
} | ||
|
||
inflight := i.inflightPushRequests.Inc() | ||
decreaseInflightInDefer := true | ||
inflightBytes := int64(0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't know the requestSize we may want to set inflightBytes
to i.inflightPushRequestsBytes.Load()
so that the later check runs at least on the inflight requests for which size was known.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, that's a great point. I'll add test for this as well, and double-check if we do the same on distributor.
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
…ytes error. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
… known. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
6472990
to
6b294ad
Compare
…works. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
What this PR does
This PR implements total request size instance limit. Works best in combination with
-ingester.limit-inflight-requests-using-grpc-method-limiter
flag, and distributors with #6490.Which issue(s) this PR fixes or relates to
Previous attempt #5958, rejected because the check is not very effective when done from inside Push handler in ingester.
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]