From 5c8815ea068cdc6cb456410492c1020467b99408 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Mon, 21 Mar 2022 14:19:30 +0100 Subject: [PATCH 1/4] Migrated query-frontend doc to Hugo bundle Signed-off-by: Marco Pracucci --- .../{query-frontend.md => query-frontend/index.md} | 2 +- .../query-frontend}/query-frontend-architecture.png | Bin .../architecture/components/query-scheduler.md | 4 ++-- docs/sources/operators-guide/reference-glossary.md | 2 +- .../operators-guide/reference-http-api/_index.md | 2 +- .../planning-capacity.md | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) rename docs/sources/operators-guide/architecture/components/{query-frontend.md => query-frontend/index.md} (98%) rename docs/sources/operators-guide/{images => architecture/components/query-frontend}/query-frontend-architecture.png (100%) diff --git a/docs/sources/operators-guide/architecture/components/query-frontend.md b/docs/sources/operators-guide/architecture/components/query-frontend/index.md similarity index 98% rename from docs/sources/operators-guide/architecture/components/query-frontend.md rename to docs/sources/operators-guide/architecture/components/query-frontend/index.md index 7d6234fb8b0..6905cc3edcf 100644 --- a/docs/sources/operators-guide/architecture/components/query-frontend.md +++ b/docs/sources/operators-guide/architecture/components/query-frontend/index.md @@ -18,7 +18,7 @@ To connect the queriers with the query-frontends, configure queriers with the qu We recommend that you run at least two query-frontend replicas for high-availability reasons. -![Query-frontend architecture](../../../images/query-frontend-architecture.png) +![Query-frontend architecture](query-frontend-architecture.png) [//]: # "Diagram source at https://docs.google.com/presentation/d/1bHp8_zcoWCYoNU2AhO2lSagQyuIrghkCncViSqn14cU/edit" diff --git a/docs/sources/operators-guide/images/query-frontend-architecture.png b/docs/sources/operators-guide/architecture/components/query-frontend/query-frontend-architecture.png similarity index 100% rename from docs/sources/operators-guide/images/query-frontend-architecture.png rename to docs/sources/operators-guide/architecture/components/query-frontend/query-frontend-architecture.png diff --git a/docs/sources/operators-guide/architecture/components/query-scheduler.md b/docs/sources/operators-guide/architecture/components/query-scheduler.md index adcdd4e7357..b3dd8ae7c61 100644 --- a/docs/sources/operators-guide/architecture/components/query-scheduler.md +++ b/docs/sources/operators-guide/architecture/components/query-scheduler.md @@ -15,7 +15,7 @@ The query-scheduler is an optional, stateless component that retains a queue of The following flow describes how a queries moves through a Grafana Mimir cluster: -1. The [query-frontend]({{< relref "./query-frontend/_index.md" >}}) receives queries, and then either splits and shards them, or serves them from the cache. +1. The [query-frontend]({{< relref "./query-frontend/index.md" >}}) receives queries, and then either splits and shards them, or serves them from the cache. 1. The query-frontend enqueues the queries into a query-scheduler. 1. The query-scheduler stores the queries in an in-memory queue where they wait for a querier to pick them up. 1. Queriers pick up the queries, and executes them. @@ -23,7 +23,7 @@ The following flow describes how a queries moves through a Grafana Mimir cluster ## Benefits of using the query-scheduler -Query-scheduler enables the scaling of query-frontends. You might experience challenges when you scale query-frontend. To learn more about query-frontend scalability limits, refer to [Why query-frontend scalability is limited]({{< relref "query-frontend/_index.md#why-query-frontend-scalability-is-limited" >}}). +Query-scheduler enables the scaling of query-frontends. You might experience challenges when you scale query-frontend. To learn more about query-frontend scalability limits, refer to [Why query-frontend scalability is limited]({{< relref "query-frontend/index.md#why-query-frontend-scalability-is-limited" >}}). ### How query-scheduler solves query-frontend scalability limits diff --git a/docs/sources/operators-guide/reference-glossary.md b/docs/sources/operators-guide/reference-glossary.md index cf7abd89ad6..6f9932952cf 100644 --- a/docs/sources/operators-guide/reference-glossary.md +++ b/docs/sources/operators-guide/reference-glossary.md @@ -33,7 +33,7 @@ For component specific documentation, refer to one of the following topics: - [Compactor]({{< relref "architecture/components/compactor.md" >}}) - [Distributor]({{< relref "architecture/components/distributor.md" >}}) - [Ingester]({{< relref "architecture/components/ingester.md" >}}) -- [Query-frontend]({{< relref "architecture/components/query-frontend/_index.md" >}}) +- [Query-frontend]({{< relref "architecture/components/query-frontend/index.md" >}}) - [Query-scheduler]({{< relref "architecture/components/query-scheduler.md" >}}) - [Store-gateway]({{< relref "architecture/components/store-gateway.md" >}}) - [Optional: Alertmanager]({{< relref "architecture/components/alertmanager.md" >}}) diff --git a/docs/sources/operators-guide/reference-http-api/_index.md b/docs/sources/operators-guide/reference-http-api/_index.md index c0c26353a91..11261b01cd2 100644 --- a/docs/sources/operators-guide/reference-http-api/_index.md +++ b/docs/sources/operators-guide/reference-http-api/_index.md @@ -305,7 +305,7 @@ This endpoint displays a web page with the ingesters hash ring status, including ## Querier / Query-frontend -The following endpoints are exposed both by the [querier]({{< relref "../architecture/components/querier.md" >}}) and [query-frontend]({{< relref "../architecture/components/query-frontend/_index.md" >}}). +The following endpoints are exposed both by the [querier]({{< relref "../architecture/components/querier.md" >}}) and [query-frontend]({{< relref "../architecture/components/query-frontend/index.md" >}}). ### Instant query diff --git a/docs/sources/operators-guide/running-production-environment/planning-capacity.md b/docs/sources/operators-guide/running-production-environment/planning-capacity.md index f9b6164b9c7..42eef93534d 100644 --- a/docs/sources/operators-guide/running-production-environment/planning-capacity.md +++ b/docs/sources/operators-guide/running-production-environment/planning-capacity.md @@ -75,7 +75,7 @@ Estimated required CPU, memory, and disk space: ### Query-frontend -The [query-frontend]({{< relref "../architecture/components/query-frontend/_index.md" >}}) component resources utilization is determined by the number of queries per second. +The [query-frontend]({{< relref "../architecture/components/query-frontend/index.md" >}}) component resources utilization is determined by the number of queries per second. Estimated required CPU and memory: From 2db34f6e4180b291e8de263f715f18ee9913eb2f Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Mon, 21 Mar 2022 14:21:55 +0100 Subject: [PATCH 2/4] Migrated query-scheduler doc to Hugo bundle Signed-off-by: Marco Pracucci --- .../architecture/components/query-frontend/index.md | 4 ++-- .../index.md} | 8 ++++---- .../query-scheduler-architecture.png | Bin docs/sources/operators-guide/reference-glossary.md | 2 +- .../planning-capacity.md | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) rename docs/sources/operators-guide/architecture/components/{query-scheduler.md => query-scheduler/index.md} (85%) rename docs/sources/operators-guide/{images => architecture/components/query-scheduler}/query-scheduler-architecture.png (100%) diff --git a/docs/sources/operators-guide/architecture/components/query-frontend/index.md b/docs/sources/operators-guide/architecture/components/query-frontend/index.md index 6905cc3edcf..db63a7a9cda 100644 --- a/docs/sources/operators-guide/architecture/components/query-frontend/index.md +++ b/docs/sources/operators-guide/architecture/components/query-frontend/index.md @@ -71,7 +71,7 @@ The query-frontend also provides [query sharding]({{< relref "query-sharding.md" The query-frontend scalability is limited by the configured number of workers per querier. -When you don't use the [query-scheduler]({{< relref "./query-scheduler.md">}}), the query-frontend stores a queue of queries to execute. +When you don't use the [query-scheduler]({{< relref "./query-scheduler/index.md">}}), the query-frontend stores a queue of queries to execute. A querier runs `-querier.max-concurrent` workers and each worker connects to one of the query-frontend replicas to pull queries to execute. A querier worker executes one query at a time. @@ -88,7 +88,7 @@ The queries exceeding the configured maximum concurrency create a backlog in the The backlog might cause a suboptimal utilization of querier resources, which can result in poor query performance when you run Grafana Mimir at scale. -The [query-scheduler]({{< relref "./query-scheduler.md" >}}) is an optional component that you can deploy to overcome the query-frontend scalability limitations. +The [query-scheduler]({{< relref "./query-scheduler/index.md" >}}) is an optional component that you can deploy to overcome the query-frontend scalability limitations. ## DNS configuration and readiness diff --git a/docs/sources/operators-guide/architecture/components/query-scheduler.md b/docs/sources/operators-guide/architecture/components/query-scheduler/index.md similarity index 85% rename from docs/sources/operators-guide/architecture/components/query-scheduler.md rename to docs/sources/operators-guide/architecture/components/query-scheduler/index.md index b3dd8ae7c61..312f04dd592 100644 --- a/docs/sources/operators-guide/architecture/components/query-scheduler.md +++ b/docs/sources/operators-guide/architecture/components/query-scheduler/index.md @@ -7,15 +7,15 @@ weight: 120 # (Optional) Grafana Mimir query-scheduler -The query-scheduler is an optional, stateless component that retains a queue of queries to execute, and distributes the workload to available [queriers]({{< relref "querier.md" >}}). +The query-scheduler is an optional, stateless component that retains a queue of queries to execute, and distributes the workload to available [queriers]({{< relref "../querier.md" >}}). -![Query-scheduler architecture](../../../images/query-scheduler-architecture.png) +![Query-scheduler architecture](query-scheduler-architecture.png) [//]: # "Diagram source at https://docs.google.com/presentation/d/1bHp8_zcoWCYoNU2AhO2lSagQyuIrghkCncViSqn14cU/edit" The following flow describes how a queries moves through a Grafana Mimir cluster: -1. The [query-frontend]({{< relref "./query-frontend/index.md" >}}) receives queries, and then either splits and shards them, or serves them from the cache. +1. The [query-frontend]({{< relref "../query-frontend/index.md" >}}) receives queries, and then either splits and shards them, or serves them from the cache. 1. The query-frontend enqueues the queries into a query-scheduler. 1. The query-scheduler stores the queries in an in-memory queue where they wait for a querier to pick them up. 1. Queriers pick up the queries, and executes them. @@ -23,7 +23,7 @@ The following flow describes how a queries moves through a Grafana Mimir cluster ## Benefits of using the query-scheduler -Query-scheduler enables the scaling of query-frontends. You might experience challenges when you scale query-frontend. To learn more about query-frontend scalability limits, refer to [Why query-frontend scalability is limited]({{< relref "query-frontend/index.md#why-query-frontend-scalability-is-limited" >}}). +Query-scheduler enables the scaling of query-frontends. You might experience challenges when you scale query-frontend. To learn more about query-frontend scalability limits, refer to [Why query-frontend scalability is limited]({{< relref "../query-frontend/index.md#why-query-frontend-scalability-is-limited" >}}). ### How query-scheduler solves query-frontend scalability limits diff --git a/docs/sources/operators-guide/images/query-scheduler-architecture.png b/docs/sources/operators-guide/architecture/components/query-scheduler/query-scheduler-architecture.png similarity index 100% rename from docs/sources/operators-guide/images/query-scheduler-architecture.png rename to docs/sources/operators-guide/architecture/components/query-scheduler/query-scheduler-architecture.png diff --git a/docs/sources/operators-guide/reference-glossary.md b/docs/sources/operators-guide/reference-glossary.md index 6f9932952cf..44c4dc908db 100644 --- a/docs/sources/operators-guide/reference-glossary.md +++ b/docs/sources/operators-guide/reference-glossary.md @@ -34,7 +34,7 @@ For component specific documentation, refer to one of the following topics: - [Distributor]({{< relref "architecture/components/distributor.md" >}}) - [Ingester]({{< relref "architecture/components/ingester.md" >}}) - [Query-frontend]({{< relref "architecture/components/query-frontend/index.md" >}}) -- [Query-scheduler]({{< relref "architecture/components/query-scheduler.md" >}}) +- [Query-scheduler]({{< relref "architecture/components/query-scheduler/index.md" >}}) - [Store-gateway]({{< relref "architecture/components/store-gateway.md" >}}) - [Optional: Alertmanager]({{< relref "architecture/components/alertmanager.md" >}}) - [Optional: Ruler]({{< relref "architecture/components/ruler.md" >}}) diff --git a/docs/sources/operators-guide/running-production-environment/planning-capacity.md b/docs/sources/operators-guide/running-production-environment/planning-capacity.md index 42eef93534d..6fbe98c6525 100644 --- a/docs/sources/operators-guide/running-production-environment/planning-capacity.md +++ b/docs/sources/operators-guide/running-production-environment/planning-capacity.md @@ -84,7 +84,7 @@ Estimated required CPU and memory: ### (Optional) Query-scheduler -The [query-scheduler]({{< relref "../architecture/components/query-scheduler.md" >}}) component resources’ utilization is determined by the number of queries per second. +The [query-scheduler]({{< relref "../architecture/components/query-scheduler/index.md" >}}) component resources’ utilization is determined by the number of queries per second. Estimated required CPU and memory: From 7c6d3deb4381f043896251c88b729db0d5d9107e Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Mon, 21 Mar 2022 14:29:28 +0100 Subject: [PATCH 3/4] Migrated compactor doc to Hugo bundle Signed-off-by: Marco Pracucci --- Makefile | 2 +- .../about-grafana-mimir-architecture.md | 4 ++-- .../operators-guide/architecture/bucket-index.md | 2 +- ...mpactor-horizontal-and-vertical-compaction.png | Bin .../compactor}/compactor-split-and-merge.png | Bin .../{compactor.md => compactor/index.md} | 14 +++++++------- .../components/query-frontend/index.md | 10 +++++----- .../operators-guide/architecture/hash-ring.md | 2 +- .../sources/operators-guide/reference-glossary.md | 2 +- .../planning-capacity.md | 4 ++-- .../production-tips.md | 6 +++--- docs/sources/release-notes/v2.0.md | 2 +- 12 files changed, 24 insertions(+), 24 deletions(-) rename docs/sources/operators-guide/{images => architecture/components/compactor}/compactor-horizontal-and-vertical-compaction.png (100%) rename docs/sources/operators-guide/{images => architecture/components/compactor}/compactor-split-and-merge.png (100%) rename docs/sources/operators-guide/architecture/components/{compactor.md => compactor/index.md} (85%) diff --git a/Makefile b/Makefile index 83f8e5a0f29..f5caf93c2f1 100644 --- a/Makefile +++ b/Makefile @@ -502,4 +502,4 @@ integration-tests: cmd/mimir/$(UPTODATE) include docs/docs.mk DOCS_DIR = docs/sources -docs: doc +docs: diff --git a/docs/sources/operators-guide/architecture/about-grafana-mimir-architecture.md b/docs/sources/operators-guide/architecture/about-grafana-mimir-architecture.md index b5e63b5c827..d8208bd9f14 100644 --- a/docs/sources/operators-guide/architecture/about-grafana-mimir-architecture.md +++ b/docs/sources/operators-guide/architecture/about-grafana-mimir-architecture.md @@ -48,9 +48,9 @@ For more information, refer to [timeline of block uploads]({{< relref "../runnin #### Series sharding and replication By default, each time series is replicated to three ingesters, and each ingester writes its own block to the long-term storage. -The [Compactor]({{< relref "components/compactor.md" >}}) merges blocks from multiple ingesters into a single block, and removes duplicate samples. +The [Compactor]({{< relref "components/compactor/index.md" >}}) merges blocks from multiple ingesters into a single block, and removes duplicate samples. Blocks compaction significantly reduces storage utilization. -For more information, refer to [Compactor]({{< relref "components/compactor.md" >}}) and [Production tips]({{< relref "../running-production-environment/production-tips.md" >}}). +For more information, refer to [Compactor]({{< relref "components/compactor/index.md" >}}) and [Production tips]({{< relref "../running-production-environment/production-tips.md" >}}). ### The read path diff --git a/docs/sources/operators-guide/architecture/bucket-index.md b/docs/sources/operators-guide/architecture/bucket-index.md index 100a19a147b..b6b1a8062f8 100644 --- a/docs/sources/operators-guide/architecture/bucket-index.md +++ b/docs/sources/operators-guide/architecture/bucket-index.md @@ -38,7 +38,7 @@ The `bucket-index.json.gz` contains: ## How it gets updated -The [compactor](./compactor.md) periodically scans the bucket and uploads an updated bucket index to the storage. +The [compactor](./compactor/index.md) periodically scans the bucket and uploads an updated bucket index to the storage. You can configure the frequency with which the bucket index is updated via `-compactor.cleanup-interval`. The use of the bucket index is optional, but the index is built and updated by the compactor even if `-blocks-storage.bucket-store.bucket-index.enabled=false`. diff --git a/docs/sources/operators-guide/images/compactor-horizontal-and-vertical-compaction.png b/docs/sources/operators-guide/architecture/components/compactor/compactor-horizontal-and-vertical-compaction.png similarity index 100% rename from docs/sources/operators-guide/images/compactor-horizontal-and-vertical-compaction.png rename to docs/sources/operators-guide/architecture/components/compactor/compactor-horizontal-and-vertical-compaction.png diff --git a/docs/sources/operators-guide/images/compactor-split-and-merge.png b/docs/sources/operators-guide/architecture/components/compactor/compactor-split-and-merge.png similarity index 100% rename from docs/sources/operators-guide/images/compactor-split-and-merge.png rename to docs/sources/operators-guide/architecture/components/compactor/compactor-split-and-merge.png diff --git a/docs/sources/operators-guide/architecture/components/compactor.md b/docs/sources/operators-guide/architecture/components/compactor/index.md similarity index 85% rename from docs/sources/operators-guide/architecture/components/compactor.md rename to docs/sources/operators-guide/architecture/components/compactor/index.md index 8c05f9fddba..5598928d8f5 100644 --- a/docs/sources/operators-guide/architecture/components/compactor.md +++ b/docs/sources/operators-guide/architecture/components/compactor/index.md @@ -12,7 +12,7 @@ The compactor increases query performance and reduces long-term storage usage by The compactor is the component responsible for: - Compacting multiple blocks of a given tenant into a single, optimized larger block. This deduplicates chunks and reduces the size of the index, resulting in reduced storage costs. Querying fewer blocks is faster, so it also increases query speed. -- Keeping the per-tenant bucket index updated. The [bucket index]({{< relref "../bucket-index.md" >}}) is used by [queriers]({{< relref "querier.md" >}}), [store-gateways]({{< relref "store-gateway.md" >}}), and [rulers]({{< relref "ruler.md" >}}) to discover both new blocks and deleted blocks in the storage. +- Keeping the per-tenant bucket index updated. The [bucket index]({{< relref "../../bucket-index.md" >}}) is used by [queriers]({{< relref "../querier.md" >}}), [store-gateways]({{< relref "../store-gateway.md" >}}), and [rulers]({{< relref "../ruler.md" >}}) to discover both new blocks and deleted blocks in the storage. - Deleting blocks that are no longer within a configurable retention period. The compactor is stateless. @@ -27,7 +27,7 @@ The compactor runs at regular, configurable intervals. **Horizontal compaction** triggers after a vertical compaction. It compacts several blocks with adjacent range periods into a single larger block. The total size of the associated block chunks does not change after horizontal compaction. The horizontal compaction may significantly reduce the size of the index and the index-header kept in memory by store-gateways. -![Compactor - horizontal and vertical compaction](../../../images/compactor-horizontal-and-vertical-compaction.png) +![Compactor - horizontal and vertical compaction](compactor-horizontal-and-vertical-compaction.png) @@ -38,7 +38,7 @@ Compaction can be tuned for clusters with large tenants. Configuration specifies - **Vertical scaling**
The setting `-compactor.compaction-concurrency` configures the max number of concurrent compactions running in a single compactor instance. Each compaction uses one CPU core. - **Horizontal scaling**
- By default, tenant blocks can be compacted by any Grafana Mimir compactor. When you enable compactor [shuffle sharding]({{< relref "../../configuring/configuring-shuffle-sharding.md" >}}) by setting `-compactor.compactor-tenant-shard-size` (or its respective YAML configuration option) to a value higher than `0` and lower than the number of available compactors, only the specified number of compactors are eligible to compact blocks for a given tenant. + By default, tenant blocks can be compacted by any Grafana Mimir compactor. When you enable compactor [shuffle sharding]({{< relref "../../../configuring/configuring-shuffle-sharding.md" >}}) by setting `-compactor.compactor-tenant-shard-size` (or its respective YAML configuration option) to a value higher than `0` and lower than the number of available compactors, only the specified number of compactors are eligible to compact blocks for a given tenant. ## Compaction algorithm @@ -53,7 +53,7 @@ To split, the first level of compaction, for example `2h`, the compactor divides The compactor merges the split blocks for each shard. This compacts all _N_ split blocks of a given shard. The merge reduces the number of blocks from _N \* M_ to _M_. For a given compaction time range, there will be a compacted block for each of the _M_ shards. -![Compactor - split-and-merge compaction strategy](../../../images/compactor-split-and-merge.png) +![Compactor - split-and-merge compaction strategy](compactor-split-and-merge.png) @@ -73,9 +73,9 @@ The compactor shards compaction jobs, either from a single tenant or multiple te Whenever the pool of compactors grows or shrinks, tenants and jobs are resharded across the available compactor instances without any manual intervention. -Compactor sharding uses a [hash ring]({{< relref "../hash-ring.md" >}}). At startup, a compactor generates random tokens and registers itself to the compactor hash ring. While running, it periodically scans the storage bucket at every interval defined by `-compactor.compaction-interval`, to discover the list of tenants in storage and compact blocks for each tenant which hash matches the token ranges assigned to the instance itself within the hash ring. +Compactor sharding uses a [hash ring]({{< relref "../../hash-ring.md" >}}). At startup, a compactor generates random tokens and registers itself to the compactor hash ring. While running, it periodically scans the storage bucket at every interval defined by `-compactor.compaction-interval`, to discover the list of tenants in storage and compact blocks for each tenant which hash matches the token ranges assigned to the instance itself within the hash ring. -To configure the compactors' hash ring, refer to [configuring hash rings]({{< relref "../../configuring/configuring-hash-rings.md" >}}). +To configure the compactors' hash ring, refer to [configuring hash rings]({{< relref "../../../configuring/configuring-hash-rings.md" >}}). ### Waiting for a stable hash ring at startup @@ -132,4 +132,4 @@ Alternatively, assuming the largest `-compactor.block-ranges` is `24h` (the defa Refer to the [compactor](../../../configuring/reference-configuration-parameters/#compactor) block section and the [limits](../../../configuring/reference-configuration-parameters/#limits) block section for details of compaction-related configuration. -The [alertmanager]({{< relref "alertmanager.md" >}}) and [ruler]({{< relref "ruler.md" >}}) components can also use object storage to store their configurations and rules uploaded by users. In that case a separate bucket should be created to store alertmanager configurations and rules: using the same bucket between ruler/alertmanager and blocks will cause issues with the compactor. +The [alertmanager]({{< relref "../alertmanager.md" >}}) and [ruler]({{< relref "../ruler.md" >}}) components can also use object storage to store their configurations and rules uploaded by users. In that case a separate bucket should be created to store alertmanager configurations and rules: using the same bucket between ruler/alertmanager and blocks will cause issues with the compactor. diff --git a/docs/sources/operators-guide/architecture/components/query-frontend/index.md b/docs/sources/operators-guide/architecture/components/query-frontend/index.md index db63a7a9cda..734f1402ddd 100644 --- a/docs/sources/operators-guide/architecture/components/query-frontend/index.md +++ b/docs/sources/operators-guide/architecture/components/query-frontend/index.md @@ -7,7 +7,7 @@ weight: 60 # Grafana Mimir query-frontend -The query-frontend is a stateless component that provides the same API as the [querier]({{< relref "./querier.md" >}}) and can be used to accelerate the read path. +The query-frontend is a stateless component that provides the same API as the [querier]({{< relref "../querier.md" >}}) and can be used to accelerate the read path. Although the query-frontend is not required, we recommend that you deploy it. When you deploy the query-frontend, you should make query requests to the query-frontend instead of the queriers. The queriers are required within the cluster to execute the queries. @@ -65,13 +65,13 @@ Although aligning the step parameter to the query time range increases the perfo ### About query sharding -The query-frontend also provides [query sharding]({{< relref "query-sharding.md" >}}). +The query-frontend also provides [query sharding]({{< relref "../../query-sharding.md" >}}). ## Why query-frontend scalability is limited The query-frontend scalability is limited by the configured number of workers per querier. -When you don't use the [query-scheduler]({{< relref "./query-scheduler/index.md">}}), the query-frontend stores a queue of queries to execute. +When you don't use the [query-scheduler]({{< relref "../query-scheduler/index.md">}}), the query-frontend stores a queue of queries to execute. A querier runs `-querier.max-concurrent` workers and each worker connects to one of the query-frontend replicas to pull queries to execute. A querier worker executes one query at a time. @@ -88,12 +88,12 @@ The queries exceeding the configured maximum concurrency create a backlog in the The backlog might cause a suboptimal utilization of querier resources, which can result in poor query performance when you run Grafana Mimir at scale. -The [query-scheduler]({{< relref "./query-scheduler/index.md" >}}) is an optional component that you can deploy to overcome the query-frontend scalability limitations. +The [query-scheduler]({{< relref "../query-scheduler/index.md" >}}) is an optional component that you can deploy to overcome the query-frontend scalability limitations. ## DNS configuration and readiness When a query-frontend starts up, it does not immediately have queriers attached to it. -The [`/ready` endpoint]({{< relref "../../reference-http-api/#readiness-probe" >}}) returns an HTTP 200 status code only when the query-frontend has at least one querier attached to it, and is then ready to serve queries. +The [`/ready` endpoint]({{< relref "../../../reference-http-api/#readiness-probe" >}}) returns an HTTP 200 status code only when the query-frontend has at least one querier attached to it, and is then ready to serve queries. Configure the `/ready` endpoint as a healthcheck in your load balancer; otherwise, a query-frontend scale-out event might result in failed queries or high latency until queriers connect to the query-frontend. If you use query-frontend with query-scheduler, the `/ready` endpoint reports an HTTP 200 status code only after the query-frontend connects to at least one query-scheduler. diff --git a/docs/sources/operators-guide/architecture/hash-ring.md b/docs/sources/operators-guide/architecture/hash-ring.md index 15b7232cde7..20622c64bd4 100644 --- a/docs/sources/operators-guide/architecture/hash-ring.md +++ b/docs/sources/operators-guide/architecture/hash-ring.md @@ -74,7 +74,7 @@ Each of the following components builds an independent hash ring: - [Ingesters]({{< relref "components/ingester.md" >}}) shard and replicate series. - [Distributors]({{< relref "components/distributor.md" >}}) enforce rate limits. -- [Compactors]({{< relref "components/compactor.md" >}}) shard compaction workload. +- [Compactors]({{< relref "components/compactor/index.md" >}}) shard compaction workload. - [Store-gateways]({{< relref "components/store-gateway.md" >}}) shard blocks to query from long-term storage. - [(Optional) Rulers]({{< relref "components/ruler.md" >}}) shard rule groups to evaluate. - [(Optional) Alertmanagers]({{< relref "components/alertmanager.md" >}}) shard tenants. diff --git a/docs/sources/operators-guide/reference-glossary.md b/docs/sources/operators-guide/reference-glossary.md index 44c4dc908db..11c794739a2 100644 --- a/docs/sources/operators-guide/reference-glossary.md +++ b/docs/sources/operators-guide/reference-glossary.md @@ -30,7 +30,7 @@ Grafana Mimir comprises several components. Each component provides a specific function to the system. For component specific documentation, refer to one of the following topics: -- [Compactor]({{< relref "architecture/components/compactor.md" >}}) +- [Compactor]({{< relref "architecture/components/compactor/index.md" >}}) - [Distributor]({{< relref "architecture/components/distributor.md" >}}) - [Ingester]({{< relref "architecture/components/ingester.md" >}}) - [Query-frontend]({{< relref "architecture/components/query-frontend/index.md" >}}) diff --git a/docs/sources/operators-guide/running-production-environment/planning-capacity.md b/docs/sources/operators-guide/running-production-environment/planning-capacity.md index 6fbe98c6525..c5ffc9c0b9e 100644 --- a/docs/sources/operators-guide/running-production-environment/planning-capacity.md +++ b/docs/sources/operators-guide/running-production-environment/planning-capacity.md @@ -130,7 +130,7 @@ The rules evaluation is computationally equal to queries execution, so the queri ### Compactor -The [compactor]({{< relref "../architecture/components/compactor.md" >}}) component resources utilization is determined by the number of active series. +The [compactor]({{< relref "../architecture/components/compactor/index.md" >}}) component resources utilization is determined by the number of active series. The compactor can scale horizontally both in Grafana Mimir clusters with one tenant and multiple tenants. We recommend to run at least one compactor instance every 20 million active series ingested in total in the Grafana Mimir cluster, calculated before ingesters replication. @@ -141,7 +141,7 @@ Assuming you run one compactor instance every 20 million active series, the esti - Memory: 4GB - Disk: 300GB -For more information about disk requirements, refer to [Compactor disk utilization]({{< relref "../architecture/components/compactor.md#compactor-disk-utilization" >}}). +For more information about disk requirements, refer to [Compactor disk utilization]({{< relref "../architecture/components/compactor/index.md#compactor-disk-utilization" >}}). **To estimate the number of active series before ingesters replication, query the number of active series across all Prometheus servers:** diff --git a/docs/sources/operators-guide/running-production-environment/production-tips.md b/docs/sources/operators-guide/running-production-environment/production-tips.md index 18d6e22cdc3..75d0605438e 100644 --- a/docs/sources/operators-guide/running-production-environment/production-tips.md +++ b/docs/sources/operators-guide/running-production-environment/production-tips.md @@ -48,7 +48,7 @@ When running Grafana Mimir at scale, querying non-compacted blocks might be inef Configure Grafana Mimir to ensure only compacted blocks are queried: -1. Configure compactor's `-compactor.split-and-merge-shards` and `-compactor.split-groups` for every tenant with more than 20 million active series. For more information about configuring the compactor's split and merge shards, refer to [compactor]({{< relref "../architecture/components/compactor.md" >}}). +1. Configure compactor's `-compactor.split-and-merge-shards` and `-compactor.split-groups` for every tenant with more than 20 million active series. For more information about configuring the compactor's split and merge shards, refer to [compactor]({{< relref "../architecture/components/compactor/index.md" >}}). 1. Configure querier's `-querier.query-store-after` equal to `-querier.query-ingesters-within` minus five minutes. The five-minute delta is recommended to ensure the time range on the boundary is queried both from ingesters and queriers. #### How to estimate `-querier.query-store-after` @@ -61,7 +61,7 @@ The following diagram shows all of the timings involved in the estimation. This - The compactor takes up to three hours to compact two-hour blocks shipped from all ingesters - Querier and store-gateways take up to 15 minutes to discover and load a new compacted block -Based on these assumptions, in the worst-case scenario, it takes up to six hours and 45 minutes from when a sample is ingested until that sample has been appended to a block flushed to the storage and the block is [vertically compacted](./compactor.md) with all other overlapping two-hour blocks shipped from ingesters. +Based on these assumptions, in the worst-case scenario, it takes up to six hours and 45 minutes from when a sample is ingested until that sample has been appended to a block flushed to the storage and the block is [vertically compacted](./compactor/index.md) with all other overlapping two-hour blocks shipped from ingesters. ![Avoid querying non compacted blocks](../../images/avoid-querying-non-compacted-blocks.png) @@ -89,7 +89,7 @@ We recommend configuring the system's `file-max` ulimit at least to `65536` to a ### Ensure the compactor has enough disk space The compactor requires a lot of disk space to download source blocks from the long-term storage and temporarily store the compacted block before uploading it to the storage. -For more information about required disk space, refer to [Compactor disk utilization](../architecture/components/compactor.md#compactor-disk-utilization). +For more information about required disk space, refer to [Compactor disk utilization](../architecture/components/compactor/index.md#compactor-disk-utilization). ## Caching diff --git a/docs/sources/release-notes/v2.0.md b/docs/sources/release-notes/v2.0.md index 4aab8f4d7fe..d4da83ad329 100644 --- a/docs/sources/release-notes/v2.0.md +++ b/docs/sources/release-notes/v2.0.md @@ -34,7 +34,7 @@ These features and enhancements distinguish Grafana Mimir from Cortex 1.10.0: - **Configuration parameter reduction and classification**: We removed 36% of the configuration parameters in Grafana Mimir. All remaining configuration parameters have been classified as basic, advanced, or experimental. This is meant to make Grafana Mimir’s configuration more approachable for new users. In a default installation, you can focus exclusively on basic configuration. As you become more familiar with Grafana Mimir and want to push your clusters further, you can choose to tune advanced parameters or use experimental parameters. Refer to [parameter categories]({{< relref "../operators-guide/configuring/reference-configuration-parameters/#parameter-categories" >}}) to learn more. -- **More-scalable metrics compaction**: Grafana Mimir's new compactor uses a split-and-merge compaction algorithm. This new compactor parallelizes the compaction of overlapping blocks across multiple machines, and splits blocks in such a way as to overcome Prometheus’ [64 GB index size limit](https://grafana.com/blog/2019/10/31/lifting-the-index-size-limit-of-prometheus-with-postings-compression/). This solves a problem that we've seen with tenants with greater than 30M time series in Cortex - namely, that the count of uncompacted blocks grew indefinitely because the compactor couldn’t work fast enough. With its new compactor, a single Grafana Mimir cluster can support over one billion active time series. Refer to [compactor]({{< relref "../operators-guide/architecture/components/compactor.md" >}}) to learn more. +- **More-scalable metrics compaction**: Grafana Mimir's new compactor uses a split-and-merge compaction algorithm. This new compactor parallelizes the compaction of overlapping blocks across multiple machines, and splits blocks in such a way as to overcome Prometheus’ [64 GB index size limit](https://grafana.com/blog/2019/10/31/lifting-the-index-size-limit-of-prometheus-with-postings-compression/). This solves a problem that we've seen with tenants with greater than 30M time series in Cortex - namely, that the count of uncompacted blocks grew indefinitely because the compactor couldn’t work fast enough. With its new compactor, a single Grafana Mimir cluster can support over one billion active time series. Refer to [compactor]({{< relref "../operators-guide/architecture/components/compactor/index.md" >}}) to learn more. - **Query sharding for improved query speeds**: Grafana Mimir introduces query sharding to accelerate the execution of high-cardinality or CPU-intensive queries. Query sharding distributes the execution of a single query across multiple machines, to significantly reduce query execution time. We have seen speedups of 10 to 30x in our Grafana Cloud Metrics clusters. Refer to [query sharding]({{< relref "../operators-guide/architecture/query-sharding.md" >}}) to learn more. From 0e3708090100d5411a4751d44fd81c1b64c9be88 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Mon, 21 Mar 2022 16:22:19 +0100 Subject: [PATCH 4/4] Reverted unintended change in Makefile Signed-off-by: Marco Pracucci --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index f5caf93c2f1..83f8e5a0f29 100644 --- a/Makefile +++ b/Makefile @@ -502,4 +502,4 @@ integration-tests: cmd/mimir/$(UPTODATE) include docs/docs.mk DOCS_DIR = docs/sources -docs: +docs: doc