Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store-gateway: enable lazy loading concurrency by default #6004

Merged
merged 4 commits into from
Sep 13, 2023
Merged

store-gateway: enable lazy loading concurrency by default #6004

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8f40dda
store-gateway: enable lazy loading concurrency by default
dimitarvdimitrov Sep 12, 2023
9cd2e88
Add CHANGELOG.md entry
dimitarvdimitrov Sep 12, 2023
f731fb2
Update tests
dimitarvdimitrov Sep 12, 2023
970ca55
Update tests
dimitarvdimitrov Sep 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

### Grafana Mimir

* [CHANGE] Store-gateway: lazy-loading concurrency limit default value is now 4. #6004
* [ENHANCEMENT] Query-scheduler: add `cortex_query_scheduler_enqueue_duration_seconds` metric that records the time taken to enqueue or reject a query request. #5879
* [ENHANCEMENT] Query-frontend: add `cortex_query_frontend_enqueue_duration_seconds` metric that records the time taken to enqueue or reject a query request when not using the query-scheduler. #5879
* [ENHANCEMENT] Expose `/sync/mutex/wait/total:seconds` Go runtime metric as `go_sync_mutex_wait_total_seconds_total` from all components. #5879
Expand Down
2 changes: 1 addition & 1 deletion cmd/mimir/config-descriptor.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7710,7 +7710,7 @@
"required": false,
"desc": "Maximum number of concurrent index header loads across all tenants. If set to 0, concurrency is unlimited.",
"fieldValue": null,
"fieldDefaultValue": 0,
"fieldDefaultValue": 4,
"fieldFlag": "blocks-storage.bucket-store.index-header.lazy-loading-concurrency",
"fieldType": "int",
"fieldCategory": "experimental"
Expand Down
2 changes: 1 addition & 1 deletion cmd/mimir/help-all.txt.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,7 @@ Usage of ./cmd/mimir/mimir:
-blocks-storage.bucket-store.index-header.eager-loading-startup-enabled
[experimental] If enabled, store-gateway will periodically persist block IDs of lazy loaded index-headers and load them eagerly during startup. It is not valid to enable this if index-header lazy loading is disabled.
-blocks-storage.bucket-store.index-header.lazy-loading-concurrency int
[experimental] Maximum number of concurrent index header loads across all tenants. If set to 0, concurrency is unlimited.
[experimental] Maximum number of concurrent index header loads across all tenants. If set to 0, concurrency is unlimited. (default 4)
-blocks-storage.bucket-store.index-header.lazy-loading-enabled
If enabled, store-gateway will lazy load an index-header only once required by a query. (default true)
-blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout duration
Expand Down
2 changes: 1 addition & 1 deletion docs/sources/mimir/references/configuration-parameters/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3487,7 +3487,7 @@ bucket_store:
# (experimental) Maximum number of concurrent index header loads across all
# tenants. If set to 0, concurrency is unlimited.
# CLI flag: -blocks-storage.bucket-store.index-header.lazy-loading-concurrency
[lazy_loading_concurrency: <int> | default = 0]
[lazy_loading_concurrency: <int> | default = 4]

# (experimental) If enabled, store-gateway will persist a sparse version of
# the index-header to disk on construction and load sparse index-headers
Expand Down
4 changes: 4 additions & 0 deletions pkg/storegateway/bucket_stores_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,12 @@ func TestBucketStores_InitialSync(t *testing.T) {
# HELP cortex_bucket_stores_gate_queries_concurrent_max Number of maximum concurrent queries allowed.
# TYPE cortex_bucket_stores_gate_queries_concurrent_max gauge
cortex_bucket_stores_gate_queries_concurrent_max{gate="query"} 100
cortex_bucket_stores_gate_queries_concurrent_max{gate="index_header"} 4

# HELP cortex_bucket_stores_gate_queries_in_flight Number of queries that are currently in flight.
# TYPE cortex_bucket_stores_gate_queries_in_flight gauge
cortex_bucket_stores_gate_queries_in_flight{gate="query"} 0
cortex_bucket_stores_gate_queries_in_flight{gate="index_header"} 0
`),
"cortex_bucket_store_blocks_loaded",
"cortex_bucket_store_block_loads_total",
Expand Down Expand Up @@ -249,10 +251,12 @@ func TestBucketStores_SyncBlocks(t *testing.T) {
# HELP cortex_bucket_stores_gate_queries_concurrent_max Number of maximum concurrent queries allowed.
# TYPE cortex_bucket_stores_gate_queries_concurrent_max gauge
cortex_bucket_stores_gate_queries_concurrent_max{gate="query"} 100
cortex_bucket_stores_gate_queries_concurrent_max{gate="index_header"} 4

# HELP cortex_bucket_stores_gate_queries_in_flight Number of queries that are currently in flight.
# TYPE cortex_bucket_stores_gate_queries_in_flight gauge
cortex_bucket_stores_gate_queries_in_flight{gate="query"} 0
cortex_bucket_stores_gate_queries_in_flight{gate="index_header"} 0
`),
"cortex_bucket_store_blocks_loaded",
"cortex_bucket_store_block_loads_total",
Expand Down
2 changes: 1 addition & 1 deletion pkg/storegateway/indexheader/header.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ func (cfg *Config) RegisterFlagsWithPrefix(f *flag.FlagSet, prefix string) {
f.UintVar(&cfg.MaxIdleFileHandles, prefix+"max-idle-file-handles", 1, "Maximum number of idle file handles the store-gateway keeps open for each index-header file.")
f.BoolVar(&cfg.LazyLoadingEnabled, prefix+"lazy-loading-enabled", DefaultIndexHeaderLazyLoadingEnabled, "If enabled, store-gateway will lazy load an index-header only once required by a query.")
f.DurationVar(&cfg.LazyLoadingIdleTimeout, prefix+"lazy-loading-idle-timeout", DefaultIndexHeaderLazyLoadingIdleTimeout, "If index-header lazy loading is enabled and this setting is > 0, the store-gateway will offload unused index-headers after 'idle timeout' inactivity.")
f.IntVar(&cfg.LazyLoadingConcurrency, prefix+"lazy-loading-concurrency", 0, "Maximum number of concurrent index header loads across all tenants. If set to 0, concurrency is unlimited.")
f.IntVar(&cfg.LazyLoadingConcurrency, prefix+"lazy-loading-concurrency", 4, "Maximum number of concurrent index header loads across all tenants. If set to 0, concurrency is unlimited.")
f.BoolVar(&cfg.EagerLoadingStartupEnabled, prefix+"eager-loading-startup-enabled", false, "If enabled, store-gateway will periodically persist block IDs of lazy loaded index-headers and load them eagerly during startup. It is not valid to enable this if index-header lazy loading is disabled.")
f.BoolVar(&cfg.SparsePersistenceEnabled, prefix+"sparse-persistence-enabled", false, "If enabled, store-gateway will persist a sparse version of the index-header to disk on construction and load sparse index-headers from disk instead of the whole index-header.")
f.BoolVar(&cfg.VerifyOnLoad, prefix+"verify-on-load", false, "If true, verify the checksum of index headers upon loading them (either on startup or lazily when lazy loading is enabled). Setting to true helps detect disk corruption at the cost of slowing down index header loading.")
Expand Down
Loading