Increase default timeout for index-queries cache requests #6786
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As use of the index-queries cache has increased in the store-gateway, we see more and more timeouts making requests to it. Timeouts making a request mean the connection is closed and reopened since it assumed to be "bad". This can cause a huge number of connections on nodes to end up in `TIME_WAIT`. Increase the timeout used by the index-queries cache to 450ms from the default of 200ms in jsonnet and helm. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters
force-pushed
the
56quarters/index-cache-timeout
branch
from
ef153a0
to
86d4db7
Compare
jhalterman
approved these changes
Nov 30, 2023
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
As use of the index-queries cache has increased in the store-gateway, we see more and more timeouts making requests to it. Timeouts making a request mean the connection is closed and reopened since it assumed to be "bad". This can cause a huge number of connections on nodes to end up in
TIME_WAIT.Increase the timeout used by the index-queries cache to 450ms from the default of 200ms in jsonnet and helm.
Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.