From a596e1ab42a1fb6ebe7c52dc90bb5ff5d0ca405e Mon Sep 17 00:00:00 2001 From: Jack Baldry Date: Mon, 19 Feb 2024 14:36:10 +0000 Subject: [PATCH] Replace all blockquote admonitions with the shortcode (#7420) The [`admonition` shortcode](https://grafana.com/docs/writers-toolkit/write/shortcodes/#admonition) renders its content in a blockquote with consistent styling across the website. Signed-off-by: Jack Baldry --- ...ut-anonymous-usage-statistics-reporting.md | 4 +- .../configure/about-runtime-configuration.md | 8 +- .../mimir/configure/about-tenant-ids.md | 11 ++- .../configure/configure-custom-trackers.md | 8 +- .../mimir/configure/configure-hash-rings.md | 23 +++-- ...nfigure-high-availability-deduplication.md | 34 +++++-- .../configure-object-storage-backend.md | 13 ++- ...onfigure-out-of-order-samples-ingestion.md | 7 +- .../configure-shuffle-sharding/index.md | 4 +- .../configure-zone-aware-replication.md | 21 +++-- docs/sources/mimir/get-started/_index.md | 15 ++-- .../about-grafana-mimir-architecture/index.md | 4 +- .../mimir/manage/mimir-runbooks/_index.md | 88 ++++++++++++++----- .../dashboards/ruler/index.md | 4 +- .../perform-a-rolling-update.md | 20 +++-- .../planning-capacity.md | 12 ++- .../run-production-environment/scaling-out.md | 4 +- .../authentication-and-authorization.md | 9 +- .../manage/secure/securing-alertmanager.md | 4 +- .../securing-communications-with-tls.md | 7 +- .../manage/tools/mimir-continuous-test.md | 4 +- docs/sources/mimir/manage/tools/mimirtool.md | 34 +++++-- docs/sources/mimir/manage/tools/query-tee.md | 26 ++++-- .../architecture/components/alertmanager.md | 31 +++++-- .../architecture/components/distributor.md | 24 +++-- .../architecture/components/ingester.md | 4 +- .../components/overrides-exporter.md | 7 +- .../architecture/components/querier.md | 10 ++- .../components/query-frontend/index.md | 4 +- .../components/query-scheduler/index.md | 15 +++- .../architecture/components/ruler/index.md | 23 +++-- .../architecture/components/store-gateway.md | 24 +++-- .../architecture/deployment-modes/index.md | 5 +- .../architecture/key-value-store.md | 4 +- .../mimir/references/http-api/index.md | 54 +++++++++--- docs/sources/mimir/release-notes/v2.10.md | 4 +- .../set-up/jsonnet/configure-autoscaling.md | 8 +- .../jsonnet/configure-deployment-mode.md | 5 +- .../mimir/set-up/jsonnet/configure-ruler.md | 23 +++-- docs/sources/mimir/set-up/jsonnet/deploy.md | 13 ++- .../migrate-from-consul-to-memberlist.md | 6 +- ...igrate-from-microservices-to-read-write.md | 12 +-- .../set-up/migrate/migrate-from-cortex.md | 16 +++- .../migrate-from-thanos-or-prometheus.md | 87 ++++++++++-------- 44 files changed, 533 insertions(+), 210 deletions(-) diff --git a/docs/sources/mimir/configure/about-anonymous-usage-statistics-reporting.md b/docs/sources/mimir/configure/about-anonymous-usage-statistics-reporting.md index 54cbe6f3612..f0da90a725a 100644 --- a/docs/sources/mimir/configure/about-anonymous-usage-statistics-reporting.md +++ b/docs/sources/mimir/configure/about-anonymous-usage-statistics-reporting.md @@ -57,7 +57,9 @@ When the usage statistics reporting is enabled, Grafana Mimir collects the follo - Metadata query. - Cardinality analysis query. -> **Note**: Mimir maintainers commit to keeping the list of tracked information updated over time, and reporting any change both via the CHANGELOG and the release notes. +{{< admonition type="note" >}} +Mimir maintainers commit to keeping the list of tracked information updated over time, and reporting any change both via the CHANGELOG and the release notes. +{{< /admonition >}} ## Disable the anonymous usage statistics reporting diff --git a/docs/sources/mimir/configure/about-runtime-configuration.md b/docs/sources/mimir/configure/about-runtime-configuration.md index cfe596cde8e..94fea5df357 100644 --- a/docs/sources/mimir/configure/about-runtime-configuration.md +++ b/docs/sources/mimir/configure/about-runtime-configuration.md @@ -99,4 +99,10 @@ An advanced runtime configuration option controls if ingesters transfer encoded The parameter `ingester_stream_chunks_when_using_blocks` might only be used in runtime configuration. A value of `true` transfers encoded chunks, and a value of `false` transfers decoded series. -> **Note:** The parameter `ingester_stream_chunks_when_using_blocks` in the runtime configuration can override the CLI flag `-ingester.stream-chunks-when-using-blocks`, which is enabled by default. We strongly recommend to keep it enabled, except in rare cases where you observe rules evaluation slowing down. +{{< admonition type="note" >}} +By default, `-ingester.stream-chunks-when-using-blocks` is `true` which enables transfer of encoded chunks. + +In runtime configuration, the parameter `ingester_stream_chunks_when_using_blocks` overrides the CLI flag `-ingester.stream-chunks-when-using-blocks`. + +It's strongly recommended that you keep the transfer of encoded chunks enabled, except in rare cases where you observe rules evaluation slowing down. +{{< /admonition >}} diff --git a/docs/sources/mimir/configure/about-tenant-ids.md b/docs/sources/mimir/configure/about-tenant-ids.md index 1e9255f18f4..468b356c1b4 100644 --- a/docs/sources/mimir/configure/about-tenant-ids.md +++ b/docs/sources/mimir/configure/about-tenant-ids.md @@ -30,7 +30,12 @@ Tenant IDs must be less-than or equal-to 150 bytes or characters in length and c - Open parenthesis (`(`) - Close parenthesis (`)`) -> **Note:** For security reasons, `.` and `..` are not valid tenant IDs. -> **Note:** The tenant ID `__mimir_cluster` is unsupported because its name is used internally by Mimir. +{{< admonition type="note" >}} +For security reasons, `.` and `..` aren't valid tenant IDs. +{{< /admonition >}} -All other characters, including slashes and whitespace, are not supported. +{{< admonition type="note" >}} +`__mimir_cluster` isn't a valid tenant ID because Mimir uses the name internally. +{{< /admonition >}} + +All other characters, including slashes and whitespace, aren't supported. diff --git a/docs/sources/mimir/configure/configure-custom-trackers.md b/docs/sources/mimir/configure/configure-custom-trackers.md index 9cc6c08edcf..a2598599592 100644 --- a/docs/sources/mimir/configure/configure-custom-trackers.md +++ b/docs/sources/mimir/configure/configure-custom-trackers.md @@ -62,4 +62,10 @@ cortex_ingester_active_series_custom_tracker{name="service2", user="tenant_with_ To set up runtime overrides, refer to [runtime configuration]({{< relref "./about-runtime-configuration" >}}). -> **Note:** The custom active series trackers are exposed on each ingester. To understand the count of active series matching a particular label pattern in your Grafana Mimir cluster at a global level, you must collect and sum this metric across all ingesters. If you're running Grafana Mimir with a `replication_factor` > 1, you must also adjust for the fact that the same series will be replicated `RF` times across your ingesters. +{{< admonition type="note" >}} +Each ingester exposese the custom active series trackers. + +To understand the count of active series matching a particular label pattern in your Grafana Mimir cluster at a global level, you must collect and sum this metric across all ingesters. + +If you're running Grafana Mimir with a `replication_factor` > 1, you must also adjust for the fact that Mimir replicates the same series across your ingesters replication factor times. +{{< /admonition >}} diff --git a/docs/sources/mimir/configure/configure-hash-rings.md b/docs/sources/mimir/configure/configure-hash-rings.md index 9c1bfecd9be..765b598d4dc 100644 --- a/docs/sources/mimir/configure/configure-hash-rings.md +++ b/docs/sources/mimir/configure/configure-hash-rings.md @@ -59,11 +59,19 @@ The `-memberlist.join` can be set to: The default port is `7946`. -> **Note**: At a minimum, configure one or more addresses that resolve to a consistent subset of replicas (for example, all the ingesters). +{{< admonition type="note" >}} +At a minimum, configure one or more addresses that resolve to a consistent subset of replicas (for example, all the ingesters). +{{< /admonition >}} -> **Note**: If you're running Grafana Mimir in Kubernetes, define a [headless Kubernetes Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) which resolves to the IP addresses of all Grafana Mimir pods. Then you set `-memberlist.join` to `dnssrv+..svc.cluster.local:`. +{{< admonition type="note" >}} +If you're running Grafana Mimir in Kubernetes, define a [headless Kubernetes Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) which resolves to the IP addresses of all Grafana Mimir Pods. -> **Note**: The `memberlist` backend is configured globally and can't be customized on a per-component basis. Because `memberlist` is configured globally, the `memberlist` backend differs from other supported backends, such as Consul or etcd. +Then set `-memberlist.join` to `dnssrv+..svc.cluster.local:`. +{{< /admonition >}} + +{{< admonition type="note" >}} +The `memberlist` backend is configured globally, unlike other supported backends, and can't be customized on a per-component basis. +{{< /admonition >}} Grafana Mimir supports TLS for memberlist connections between its components. For more information about TLS configuration, refer to [secure communications with TLS]({{< relref "../manage/secure/securing-communications-with-tls" >}}). @@ -159,7 +167,10 @@ You can use the following parameters to configure the multi KV store settings: - `.multi.mirror-enabled`: Whether mirroring of writes to the secondary backend store is enabled. - `.multi.mirror-timeout`: The maximum time allowed to mirror a change to the secondary backend store. -> **Note**: Grafana Mimir does not log an error if it is unable to mirror writes to the secondary backend store. The total number of errors is only tracked through the metric `cortex_multikv_mirror_write_errors_total`. +{{< admonition type="note" >}} +Grafana Mimir doesn't log an error if it's unable to mirror writes to the secondary backend store. +However, the total number of errors is tracked through the metric `cortex_multikv_mirror_write_errors_total`. +{{< /admonition >}} The multi KV primary backend and mirroring can also be configured in the [runtime configuration file]({{< relref "./about-runtime-configuration" >}}). Changes to a multi KV Store in the runtime configuration apply to _all_ components using a multi KV store. @@ -173,7 +184,9 @@ multi_kv_config: mirror_enabled: true ``` -> **Note**: The runtime configuration settings take precedence over CLI flags. +{{< admonition type="note" >}} +The runtime configuration settings take precedence over CLI flags. +{{< /admonition >}} #### Ingester migration example diff --git a/docs/sources/mimir/configure/configure-high-availability-deduplication.md b/docs/sources/mimir/configure/configure-high-availability-deduplication.md index fb56c2f7c4f..720dfe3ca41 100644 --- a/docs/sources/mimir/configure/configure-high-availability-deduplication.md +++ b/docs/sources/mimir/configure/configure-high-availability-deduplication.md @@ -24,10 +24,15 @@ that `team-1.b` sends are dropped. And if Grafana Mimir does not see any new sam If `team-1.a` goes down for more than 30 seconds, Grafana Mimir’s HA sample handling will have switched and elected `team-1.b` as the leader. The failure timeout ensures that too much data is not dropped before failover to the other replica. -> **Note:** In a scenario where the default scrape period is 15 seconds, and the timeouts in Grafana Mimir are set to the default values, -> when a leader-election failover occurs, you'll likely only lose a single scrape of data. For any query using the `rate()` function, make the rate time interval -> at least four times that of the scrape period to account for any of these failover scenarios. -> For example, with the default scrape period of 15 seconds, use a rate time-interval at least 1-minute. +{{< admonition type="note" >}} +In a scenario where the default scrape period is 15 seconds, and the timeouts in Grafana Mimir are set to the default values, +when a leader-election failover occurs, you'll likely only lose a single scrape of data. + +For any query using the `rate()` function, make the rate time interval +at least four times that of the scrape period to account for any of these failover scenarios. + +For example, with the default scrape period of 15 seconds, use a rate time-interval at least 1-minute. +{{< /admonition >}} ## Distributor high-availability (HA) tracker @@ -69,14 +74,18 @@ global: __replica__: replica2 ``` -> **Note:** The preceding labels are external labels and have nothing to do with `remote_write` configuration. +{{< admonition type="note" >}} +The preceding labels are external labels and have nothing to do with `remote_write` configuration. +{{< /admonition >}} These two label names are configurable on a per-tenant basis within Grafana Mimir. For example, if the label name of one cluster is used by some workloads, set the label name of another cluster to something else that uniquely identifies the second cluster. Set the replica label so that the value for each Prometheus cluster is unique in that cluster. -> **Note:** Grafana Mimir drops this label when ingesting data, but preserves the cluster label. This way, your timeseries won't change when replicas change. +{{< admonition type="note" >}} +Grafana Mimir drops this label when ingesting data, but preserves the cluster label. This way, your time series won't change when replicas change. +{{< /admonition >}} ### How to configure Grafana Mimir @@ -99,8 +108,13 @@ Alternatively, you can enable the HA tracker only on a per-tenant basis, keeping The HA tracker requires a key-value (KV) store to coordinate which replica is currently elected. The supported KV stores for the HA tracker are `consul` and `etcd`. -> **Note:** `memberlist` is not supported. Memberlist-based KV stores propagate updates using the Gossip protocol, which is too slow for the -> HA tracker. The result would be that different distributors might see a different Prometheus server elected as leaders at the same time. +{{< admonition type="note" >}} +`memberlist` isn't supported. + +Memberlist-based KV stores propagate updates using the Gossip protocol, which is too slow for the HA tracker. + +The result would be that different distributors might see a different Prometheus server elected as leaders at the same time. +{{< /admonition >}} The following CLI flags (and their respective YAML configuration options) are available for configuring the HA tracker KV store: @@ -118,7 +132,9 @@ Configure the default cluster and replica label names using the following CLI fl - `-distributor.ha-tracker.cluster`: Name of the label whose value uniquely identifies a Prometheus HA cluster (defaults to `cluster`). - `-distributor.ha-tracker.replica`: Name of the label whose value uniquely identifies a Prometheus replica within the HA cluster (defaults to `__replica__`). -> **Note:** The HA label names can be overridden on a per-tenant basis by setting `ha_cluster_label` and `ha_replica_label` in the overrides section of the runtime configuration. +{{< admonition type="note" >}} +The HA label names can be overridden on a per-tenant basis by setting `ha_cluster_label` and `ha_replica_label` in the overrides section of the runtime configuration. +{{< /admonition >}} #### Example configuration diff --git a/docs/sources/mimir/configure/configure-object-storage-backend.md b/docs/sources/mimir/configure/configure-object-storage-backend.md index 31f156c2f60..48f9cc546bb 100644 --- a/docs/sources/mimir/configure/configure-object-storage-backend.md +++ b/docs/sources/mimir/configure/configure-object-storage-backend.md @@ -36,16 +36,23 @@ but it is a read-only data source and can be used to provision state into those To avoid repetition, you can use the [common configuration]({{< relref "./about-configurations#common-configurations" >}}) and fill the [`common`]({{< relref "./configuration-parameters#common" >}}) configuration block or by providing the `-common.storage.*` CLI flags. -> **Note:** Blocks storage cannot be located in the same path of the same bucket as the ruler and alertmanager stores. When using the common configuration, make [`blocks_storage`]({{< relref "./configuration-parameters#blocks_storage" >}}) use either a: +{{< admonition type="note" >}} +Blocks storage can't be located in the same path of the same bucket as the ruler and Alertmanager stores. + +When using the common configuration, make [`blocks_storage`]({{< relref "./configuration-parameters#blocks_storage" >}}) use either a: - different bucket, overriding the common bucket name - storage prefix -Grafana Mimir will fail to start if you configure blocks storage to use the same bucket and storage prefix that the alertmanager or ruler store uses. +{{< /admonition >}} + +Grafana Mimir will fail to start if you configure blocks storage to use the same bucket and storage prefix that the Alertmanager or ruler store uses. Find examples of setting up the different object stores below: -> **Note**: If you're using a mixture of YAML files and CLI flags, pay attention to their [precedence logic]({{< relref "./about-configurations#common-configurations" >}}). +{{< admonition type="note" >}} +If you're using a mixture of YAML files and CLI flags, pay attention to their [precedence logic]({{< relref "./about-configurations#common-configurations" >}}). +{{< /admonition >}} ### S3 diff --git a/docs/sources/mimir/configure/configure-out-of-order-samples-ingestion.md b/docs/sources/mimir/configure/configure-out-of-order-samples-ingestion.md index e24704e8bf0..452ce4d4611 100644 --- a/docs/sources/mimir/configure/configure-out-of-order-samples-ingestion.md +++ b/docs/sources/mimir/configure/configure-out-of-order-samples-ingestion.md @@ -69,5 +69,8 @@ The moment that a new series sample arrives, Mimir needs to determine if the ser The experimental out-of-order ingestion helps fix both the issues. -> **Note:** If you're writing metrics using Prometheus remote write or the Grafana Agent, then out-of-order samples are unexpected. -> Prometheus and Grafana Agent guarantee that samples are written in-order for the same series. +{{< admonition type="note" >}} +If you're writing metrics using Prometheus remote write or Grafana Agent, then you shouldn't expect out-of-order samples. + +Prometheus and Grafana Agent guarantee that samples are written in-order for the same series. +{{< /admonition >}} diff --git a/docs/sources/mimir/configure/configure-shuffle-sharding/index.md b/docs/sources/mimir/configure/configure-shuffle-sharding/index.md index 5bc1ab60ff2..56f3eaef4a2 100644 --- a/docs/sources/mimir/configure/configure-shuffle-sharding/index.md +++ b/docs/sources/mimir/configure/configure-shuffle-sharding/index.md @@ -71,7 +71,9 @@ Grafana Mimir supports shuffle sharding in the following components: When you run Grafana Mimir with the default configuration, shuffle sharding is disabled and you need to explicitly enable it by increasing the shard size either globally or for a given tenant. -> **Note:** If the shard size value is equal to or higher than the number of available instances, for example where `-distributor.ingestion-tenant-shard-size` is higher than the number of ingesters, then shuffle sharding is disabled and all instances are used again. +{{< admonition type="note" >}} +If the shard size value is equal to or higher than the number of available instances, for example where `-distributor.ingestion-tenant-shard-size` is higher than the number of ingesters, then shuffle sharding is disabled and all instances are used again. +{{< /admonition >}} ### Guaranteed properties diff --git a/docs/sources/mimir/configure/configure-zone-aware-replication.md b/docs/sources/mimir/configure/configure-zone-aware-replication.md index 126d1f9b4b2..f847ef75369 100644 --- a/docs/sources/mimir/configure/configure-zone-aware-replication.md +++ b/docs/sources/mimir/configure/configure-zone-aware-replication.md @@ -26,11 +26,13 @@ If an outage affects a zone containing multiple replicas, data loss might occur. With zone-aware replication enabled, Grafana Mimir ensures data replication to replicas across different zones. -> **Warning:** -> Ensure that you configure deployment tooling so that it is also zone-aware. -> The deployment tooling is responsible for executing rolling updates. -> Rolling updates should only update replicas in a single zone at any given time. -> You can utilize the [Kubernetes rollout Operator](#kubernetes-operator-for-simplifying-rollouts-of-zone-aware-components) to assist with this. +{{< admonition type="warning" >}} +Ensure that you configure deployment tooling so that it's also zone-aware. +The deployment tooling is responsible for executing rolling updates. + +Rolling updates should only update replicas in a single zone at any given time. +You can utilize the [Kubernetes rollout Operator](#kubernetes-operator-for-simplifying-rollouts-of-zone-aware-components) to assist with this. +{{< /admonition >}} Grafana Mimir supports zone-aware replication for the following: @@ -81,9 +83,12 @@ When replica counts are unbalanced, zones with fewer replicas have higher resour Most cloud providers charge for inter-availability zone networking. Deploying Grafana Mimir with zone-aware replication across multiple cloud provider availability zones likely results in additional networking costs. -> **Note:** The requests that the distributors receive are usually compressed, and the requests that the distributors send to the ingesters are uncompressed by default. -> This can result in increased cross-zone bandwidth costs (because at least two ingesters will be in different availability zones). -> If this cost is a concern, you can compress those requests by setting the `-ingester.client.grpc-compression` CLI flag, or its respective YAML configuration parameter, to `snappy` or `gzip` in the distributors. +{{< admonition type="note" >}} +The requests that the distributors receive are usually compressed, and the requests that the distributors send to the ingesters are uncompressed by default. +This can result in increased cross-zone bandwidth costs because at least two ingesters will be in different availability zones. + +If this cost is a concern, you can compress those requests by setting the `-ingester.client.grpc-compression` CLI flag, or its respective YAML configuration parameter, to `snappy` or `gzip` in the distributors. +{{< /admonition >}} ## Kubernetes operator for simplifying rollouts of zone-aware components diff --git a/docs/sources/mimir/get-started/_index.md b/docs/sources/mimir/get-started/_index.md index cd5c9067e5b..51c23ca06bb 100644 --- a/docs/sources/mimir/get-started/_index.md +++ b/docs/sources/mimir/get-started/_index.md @@ -21,8 +21,11 @@ You can get started with Grafana Mimir _imperatively_ or _declaratively_: - Verify that you have installed either a [Prometheus server](https://prometheus.io/docs/prometheus/latest/installation/) or [Grafana Agent](/docs/grafana-cloud/monitor-infrastructure/integrations/agent/flow/setup/install/). - Verify that you have installed [Docker](https://docs.docker.com/engine/install/). -> **Note:** The instructions that follow help you to deploy Grafana Mimir in [Monolithic mode]({{< relref "../references/architecture/deployment-modes#monolithic-mode" >}}). -> For information about the different ways to deploy Grafana Mimir, refer to [Grafana Mimir deployment modes]({{< relref "../references/architecture/deployment-modes" >}}). +{{< admonition type="note" >}} +The instructions that follow help you to deploy Grafana Mimir in [Monolithic mode]({{< relref "../references/architecture/deployment-modes#monolithic-mode" >}}). + +For information about the different ways to deploy Grafana Mimir, refer to [Grafana Mimir deployment modes]({{< relref "../references/architecture/deployment-modes" >}}). +{{< /admonition >}} ## Download Grafana Mimir @@ -99,9 +102,11 @@ store_gateway: ``` -> **Note**: Grafana Mimir includes a system that optionally and anonymously reports non-sensitive, non-personally identifiable information about the running Mimir cluster to a remote statistics server. -> If possible and to help us understand more about how the open source community runs Mimir, we kindly ask you to keep the usage reporting feature enabled. -> To opt out, refer to [Disable the anonymous usage statistics reporting]({{< relref "../configure/about-anonymous-usage-statistics-reporting#disable-the-anonymous-usage-statistics-reporting" >}}). +{{< admonition type="note" >}} +Grafana Mimir includes a system that optionally and anonymously reports non-sensitive, non-personally identifiable information about the running Mimir cluster to a remote statistics server to help Mimir maintainers understand how the open source community runs Mimir. + +To opt out, refer to [Disable the anonymous usage statistics reporting]({{< relref "../configure/about-anonymous-usage-statistics-reporting#disable-the-anonymous-usage-statistics-reporting" >}}). +{{< /admonition >}} ## Run Grafana Mimir diff --git a/docs/sources/mimir/get-started/about-grafana-mimir-architecture/index.md b/docs/sources/mimir/get-started/about-grafana-mimir-architecture/index.md index e5865851f2c..7b414736e8e 100644 --- a/docs/sources/mimir/get-started/about-grafana-mimir-architecture/index.md +++ b/docs/sources/mimir/get-started/about-grafana-mimir-architecture/index.md @@ -65,7 +65,9 @@ Queries coming into Grafana Mimir arrive at the [query-frontend]({{< relref "../ The query-frontend next checks the results cache. If the result of a query has been cached, the query-frontend returns the cached results. Queries that cannot be answered from the results cache are put into an in-memory queue within the query-frontend. -> **Note:** If you run the optional [query-scheduler]({{< relref "../../references/architecture/components/query-scheduler" >}}) component, this queue is maintained in the query-scheduler instead of the query-frontend. +{{< admonition type="note" >}} +If you run the optional [query-scheduler]({{< relref "../../references/architecture/components/query-scheduler" >}}) component, the query-schedule maintains the queue instead of the query-frontend. +{{< /admonition >}} The queriers act as workers, pulling queries from the queue. diff --git a/docs/sources/mimir/manage/mimir-runbooks/_index.md b/docs/sources/mimir/manage/mimir-runbooks/_index.md index d042cbf93d9..e016c42b549 100644 --- a/docs/sources/mimir/manage/mimir-runbooks/_index.md +++ b/docs/sources/mimir/manage/mimir-runbooks/_index.md @@ -1317,69 +1317,89 @@ These error IDs allow you to read related details in the documentation that foll This non-critical error occurs when Mimir receives a write request that contains a series without a metric name. Each series must have a metric name. Rarely it does not, in which case there might be a bug in the sender client. -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-metric-name-invalid This non-critical error occurs when Mimir receives a write request that contains a series with an invalid metric name. A metric name can only contain characters as defined by Prometheus’ [Metric names and labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-max-label-names-per-series This non-critical error occurs when Mimir receives a write request that contains a series with a number of labels that exceed the configured limit. The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-label-names-per-series` option. -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-max-native-histogram-buckets This non-critical error occurs when Mimir receives a write request that contains a sample that is a native histogram that has too many observation buckets. The limit protects the system from using too much memory. To configure the limit on a per-tenant basis, use the `-validation.max-native-histogram-buckets` option. -> **Note:** The series containing such samples are skipped during ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +The series containing such samples are skipped during ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-not-reducible-native-histogram This non-critical error occurs when Mimir receives a write request that contains a sample that is a native histogram that has too many observation buckets and it is not possible to reduce the buckets further. Since native buckets at the lowest resolution of -4 can cover all 64 bit float observations with a handful of buckets, this indicates that the `-validation.max-native-histogram-buckets` option is set too low (<20). -> **Note:** The series containing such samples are skipped during ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +The series containing such samples are skipped during ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-invalid-native-histogram-schema This non-critical error occurs when Mimir receives a write request that contains a sample that is a native histogram with an invalid schema number. Currently, valid schema numbers are from the range [-4, 8]. -> **Note:** The series containing such samples are skipped during ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +The series containing such samples are skipped during ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-label-invalid This non-critical error occurs when Mimir receives a write request that contains a series with an invalid label name. A label name name can only contain characters as defined by Prometheus’ [Metric names and labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-label-name-too-long This non-critical error occurs when Mimir receives a write request that contains a series with a label name whose length exceeds the configured limit. The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-length-label-name` option. -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-label-value-too-long This non-critical error occurs when Mimir receives a write request that contains a series with a label value whose length exceeds the configured limit. The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-length-label-value` option. -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-duplicate-label-names This non-critical error occurs when Mimir receives a write request that contains a series with the same label name two or more times. A series that contains a duplicated label name is invalid and gets skipped during the ingestion. -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-labels-not-sorted @@ -1387,7 +1407,9 @@ This error occurs when Mimir receives a write request that contains a series who However, Mimir internally sorts labels for series that it receives, so this error should not occur in practice. If you experience this error, [open an issue in the Mimir repository](https://github.com/grafana/mimir/issues). -> **Note:** Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< admonition type="note" >}} +Invalid series are skipped during the ingestion, and valid series within the same request are ingested. +{{< /admonition >}} ### err-mimir-too-far-in-future @@ -1395,7 +1417,9 @@ This non-critical error occurs when Mimir receives a write request that contains Mimir accepts timestamps that are slightly in the future, due to skewed clocks for example. It rejects timestamps that are too far in the future, based on the definition that you can set via the `-validation.create-grace-period` option. On a per-tenant basis, you can fine tune the tolerance by configuring the `creation_grace_period` option. -> **Note:** Only series with invalid samples are skipped during the ingestion. Valid samples within the same request are still ingested. +{{< admonition type="note" >}} +Only series with invalid samples are skipped during the ingestion. Valid samples within the same request are still ingested. +{{< /admonition >}} ### err-mimir-exemplar-too-far-in-future @@ -1403,49 +1427,63 @@ This non-critical error occurs when Mimir receives a write request that contains Mimir accepts timestamps that are slightly in the future, due to skewed clocks for example. It rejects timestamps that are too far in the future, based on the definition that you can set via the `-validation.create-grace-period` option. On a per-tenant basis, you can fine tune the tolerance by configuring the `creation_grace_period` option. -> **Note:** Only series with invalid samples are skipped during the ingestion. Valid samples within the same request are still ingested. +{{< admonition type="note" >}} +Only series with invalid samples are skipped during the ingestion. Valid samples within the same request are still ingested. +{{< /admonition >}} ### err-mimir-exemplar-labels-missing This non-critical error occurs when Mimir receives a write request that contains an exemplar without a label that identifies the related metric. An exemplar must have at least one valid label pair, otherwise it cannot be associated with any metric. -> **Note:** Invalid exemplars are skipped during the ingestion, and valid exemplars within the same request are ingested. +{{< admonition type="note" >}} +Invalid exemplars are skipped during the ingestion, and valid exemplars within the same request are ingested. +{{< /admonition >}} ### err-mimir-exemplar-labels-too-long This non-critical error occurs when Mimir receives a write request that contains an exemplar where the combined set size of its labels exceeds the limit. The limit is used to protect the system’s stability from potential abuse or mistakes, and it cannot be configured. -> **Note:** Invalid exemplars are skipped during the ingestion, and valid exemplars within the same request are ingested. +{{< admonition type="note" >}} +Invalid exemplars are skipped during the ingestion, and valid exemplars within the same request are ingested. +{{< /admonition >}} ### err-mimir-exemplar-timestamp-invalid This non-critical error occurs when Mimir receives a write request that contains an exemplar without a timestamp. An exemplar must have a valid timestamp, otherwise it cannot be correlated to any point in time. -> **Note:** Invalid exemplars are skipped during the ingestion, and valid exemplars within the same request are ingested. +{{< admonition type="note" >}} +Invalid exemplars are skipped during the ingestion, and valid exemplars within the same request are ingested. +{{< /admonition >}} ### err-mimir-metadata-missing-metric-name This non-critical error occurs when Mimir receives a write request that contains a metric metadata without a metric name. Each metric metadata must have a metric name. Rarely it does not, in which case there might be a bug in the sender client. -> **Note:** Invalid metrics metadata are skipped during the ingestion, and valid metadata within the same request are ingested. +{{< admonition type="note" >}} +Invalid metrics metadata are skipped during the ingestion, and valid metadata within the same request are ingested. +{{< /admonition >}} ### err-mimir-metric-name-too-long This non-critical error occurs when Mimir receives a write request that contains a metric metadata with a metric name whose length exceeds the configured limit. The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-metadata-length` option. -> **Note:** Invalid metrics metadata are skipped during the ingestion, and valid metadata within the same request are ingested. +{{< admonition type="note" >}} +Invalid metrics metadata are skipped during the ingestion, and valid metadata within the same request are ingested. +{{< /admonition >}} ### err-mimir-unit-too-long This non-critical error occurs when Mimir receives a write request that contains a metric metadata with unit name whose length exceeds the configured limit. The limit protects the system’s stability from potential abuse or mistakes. To configure the limit on a per-tenant basis, use the `-validation.max-metadata-length` option. -> **Note:** Invalid metrics metadata are skipped during the ingestion, and valid metadata within the same request are ingested. +{{< admonition type="note" >}} +Invalid metrics metadata are skipped during the ingestion, and valid metadata within the same request are ingested. +{{< /admonition >}} ### err-mimir-distributor-max-ingestion-rate @@ -1761,7 +1799,9 @@ How it **works**: - If the incoming timestamp is more than 1 hour older than the most recent timestamp ingested for the tenant, the sample will be rejected. -> **Note:** If the out-of-order sample ingestion is enabled, then this error is similar to `err-mimir-sample-out-of-order` below with a difference that the sample is older than the out-of-order time window as it relates to the latest sample for that particular time series or the TSDB. +{{< admonition type="note" >}} +If the out-of-order sample ingestion is enabled, then this error is similar to `err-mimir-sample-out-of-order` below with a difference that the sample is older than the out-of-order time window as it relates to the latest sample for that particular time series or the TSDB. +{{< /admonition >}} ### err-mimir-sample-out-of-order @@ -1780,7 +1820,9 @@ Common **causes**: - A Prometheus instance was restarted, and it pushed all data from its Write-Ahead Log to remote write upon restart, some of which has already been pushed and ingested. This is normal and can be ignored. - Prometheus and Mimir have the same recording rule, which generates the exact same series in both places and causes either the remote write or the rule evaluation to fail randomly, depending on timing. -> **Note:** You can learn more about out of order samples in Prometheus, in the blog post [Debugging out of order samples](https://www.robustperception.io/debugging-out-of-order-samples/). +{{< admonition type="note" >}} +You can learn more about out of order samples in Prometheus, in the blog post [Debugging out of order samples](https://www.robustperception.io/debugging-out-of-order-samples/). +{{< /admonition >}} ### err-mimir-sample-duplicate-timestamp @@ -2091,7 +2133,9 @@ Step 3: Run the `undelete-block-gcs` tool to recover the deleted blocks: cat deleted-list | undelete-block-gcs -concurrency 16 ``` -> **Note**: we recommend to try the `undelete-block-gcs` on a single block first, ensure that it gets recovered correctly and then run it against a bigger set of blocks to recover. +{{< admonition type="note" >}} +we recommend to try the `undelete-block-gcs` on a single block first, ensure that it gets recovered correctly and then run it against a bigger set of blocks to recover. +{{< /admonition >}} #### Recover accidentally deleted blocks using `gsutil` diff --git a/docs/sources/mimir/manage/monitor-grafana-mimir/dashboards/ruler/index.md b/docs/sources/mimir/manage/monitor-grafana-mimir/dashboards/ruler/index.md index 47ea1fbc46f..ddc357a463b 100644 --- a/docs/sources/mimir/manage/monitor-grafana-mimir/dashboards/ruler/index.md +++ b/docs/sources/mimir/manage/monitor-grafana-mimir/dashboards/ruler/index.md @@ -19,7 +19,9 @@ The following example shows a Ruler dashboard from a demo cluster. ![Grafana Mimir ruler dashboard](mimir-ruler.png) -> **Note:** Even while operating in [Remote ruler mode]({{< relref "../../../../references/architecture/components/ruler#remote" >}}) you will still see values for the `Read from ingesters - QPS`. +{{< admonition type="note" >}} +Even while operating in [Remote ruler mode]({{< relref "../../../../references/architecture/components/ruler#remote" >}}) there are still values for the `Read from ingesters - QPS`. +{{< /admonition >}} This is because the metrics are inclusive of intermediate services and are showing the requests that ultimately reach the ingesters. diff --git a/docs/sources/mimir/manage/run-production-environment/perform-a-rolling-update.md b/docs/sources/mimir/manage/run-production-environment/perform-a-rolling-update.md index b388fead7ee..1961bcdddbf 100644 --- a/docs/sources/mimir/manage/run-production-environment/perform-a-rolling-update.md +++ b/docs/sources/mimir/manage/run-production-environment/perform-a-rolling-update.md @@ -20,7 +20,9 @@ rolling update results in no downtime to Grafana Mimir. When you run Grafana Mimir in monolithic mode, roll out changes to one instance at a time. After you apply changes to an instance, and the instance restarts, its `/ready` endpoint returns HTTP status code `200`, which means that you can proceed with rolling out changes to another instance. -> **Note**: When you run Grafana Mimir on Kubernetes, to roll out changes to one instance at a time, configure the `Deployment` or `StatefulSet` update strategy to `RollingUpdate` and `maxUnavailable` to `1`. +{{< admonition type="note" >}} +When you run Grafana Mimir on Kubernetes, to roll out changes to one instance at a time, configure the `Deployment` or `StatefulSet` update strategy to `RollingUpdate` and `maxUnavailable` to `1`. +{{< /admonition >}} ## Microservices mode @@ -32,7 +34,9 @@ Stateful components have the following restrictions: - Ingesters: Roll out changes to one ingester at a time. - Store-gateways: Roll out changes to a maximum of two store-gateways at a time. -> **Note**: If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for a component, you can roll out changes to all component instances in the same zone at the same time. +{{< admonition type="note" >}} +If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for a component, you can roll out changes to all component instances in the same zone at the same time. +{{< /admonition >}} ### Alertmanagers @@ -44,7 +48,9 @@ Alerts notification and visualization succeed when each tenant has at least one To ensure no alerts notification, reception, or visualization fail during a rolling update, roll out changes to a maximum of two Alertmanagers at a time. -> **Note**: If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for Alertmanager, you can roll out changes to all Alertmanagers in one zone at the same time. +{{< admonition type="note" >}} +If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for Alertmanager, you can roll out changes to all Alertmanagers in one zone at the same time. +{{< /admonition >}} ### Ingesters @@ -57,7 +63,9 @@ Because series are sharded across all ingesters, Grafana Mimir tolerates up to o To ensure no query fails during a rolling update, roll out changes to one ingester at a time. -> **Note**: If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for ingesters, you can roll out changes to all ingesters in one zone at the same time. +{{< admonition type="note" >}} +If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for ingesters, you can roll out changes to all ingesters in one zone at the same time. +{{< /admonition >}} ### Store-gateways @@ -67,4 +75,6 @@ Queries succeed when each required block is loaded by at least one store-gateway To ensure no query fails during a rolling update, roll out changes to a maximum of two store-gateways at a time. -> **Note**: If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for store-gateways, you can roll out changes to all store-gateways in one zone at the same time. +{{< admonition type="note" >}} +If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for store-gateways, you can roll out changes to all store-gateways in one zone at the same time. +{{< /admonition >}} diff --git a/docs/sources/mimir/manage/run-production-environment/planning-capacity.md b/docs/sources/mimir/manage/run-production-environment/planning-capacity.md index 8940acedb03..487510afbd8 100644 --- a/docs/sources/mimir/manage/run-production-environment/planning-capacity.md +++ b/docs/sources/mimir/manage/run-production-environment/planning-capacity.md @@ -103,7 +103,9 @@ Estimated required CPU and memory: - CPU: 1 core for every 10 queries per second - Memory: 1GB for every 10 queries per second -> **Note:** The estimate is 1 CPU core and 1GB per query, with an average query latency of 100ms. +{{< admonition type="note" >}} +The estimate is 1 CPU core and 1GB per query, with an average query latency of 100ms. +{{< /admonition >}} ### Store-gateway @@ -115,9 +117,13 @@ Estimated required CPU, memory, and disk space: - Memory: 1GB every 10 queries per second - Disk: 13GB every 1 million active series -> **Note:** The CPU and memory requirements are computed by estimating 1 CPU core and 1GB per query, an average query latency of 1s when reaching the store-gateway, and only 10% of queries reaching the store-gateway. +{{< admonition type="note" >}} +The CPU and memory requirements are computed by estimating 1 CPU core and 1GB per query, an average query latency of 1s when reaching the store-gateway, and only 10% of queries reaching the store-gateway. +{{< /admonition >}} -> **Note**: The disk requirement has been estimated assuming 2 bytes per sample for compacted blocks (both index and chunks), the index-header being 0.10% of a block size, a scrape interval of 15 seconds, a retention of 1 year and store-gateways replication factor configured to 3. The resulting estimated store-gateway disk space for one series is 13KB. +{{< admonition type="note" >}} +The disk requirement has been estimated assuming 2 bytes per sample for compacted blocks (both index and chunks), the index-header being 0.10% of a block size, a scrape interval of 15 seconds, a retention of 1 year and store-gateways replication factor configured to 3. The resulting estimated store-gateway disk space for one series is 13KB. +{{< /admonition >}} **How to estimate the number of active series before ingesters replication:** diff --git a/docs/sources/mimir/manage/run-production-environment/scaling-out.md b/docs/sources/mimir/manage/run-production-environment/scaling-out.md index 901cdfaa48b..6d98489a2cb 100644 --- a/docs/sources/mimir/manage/run-production-environment/scaling-out.md +++ b/docs/sources/mimir/manage/run-production-environment/scaling-out.md @@ -51,7 +51,9 @@ Consider the following guidelines when you scale down Alertmanagers: - Scale down no more than two Alertmanagers at the same time. - Ensure at least `-alertmanager.sharding-ring.replication-factor` Alertmanager instances are running (three when running Grafana Mimir with the default configuration). -> **Note:** If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for Alertmanagers, you can, in parallel, scale down any number of Alertmanager instances within one zone at a time. +{{< admonition type="note" >}} +If you enabled [zone-aware replication]({{< relref "../../configure/configure-zone-aware-replication" >}}) for Alertmanagers, you can, in parallel, scale down any number of Alertmanager instances within one zone at a time. +{{< /admonition >}} ### Scaling down ingesters diff --git a/docs/sources/mimir/manage/secure/authentication-and-authorization.md b/docs/sources/mimir/manage/secure/authentication-and-authorization.md index 19d455235ef..973abb5e1df 100644 --- a/docs/sources/mimir/manage/secure/authentication-and-authorization.md +++ b/docs/sources/mimir/manage/secure/authentication-and-authorization.md @@ -51,7 +51,9 @@ headers: In trusted environments where you want to split series on Prometheus labels, you can run [cortex-tenant](https://github.com/blind-oracle/cortex-tenant) between a Prometheus server and Grafana Mimir. -> **Note:** cortex-tenant is a third-party community project that is not maintained by Grafana Labs. +{{< admonition type="note" >}} +cortex-tenant is a third-party community project that's not maintained by Grafana Labs. +{{< /admonition >}} When proxying the timeseries to Grafana Mimir, you can configure cortex-tenant to use specified labels as the `X-Scope-OrgID` header. @@ -67,4 +69,7 @@ After you disable multi-tenancy, Grafana Mimir components internally set the ten To set an alternative tenant ID, use the `-auth.no-auth-tenant` flag. -> **Note**: Not all tenant IDs are valid. For more information about tenant ID restrictions, refer to [About tenant IDs]({{< relref "../../configure/about-tenant-ids" >}}). +{{< admonition type="note" >}} +Not all tenant IDs are valid. +For more information about tenant ID restrictions, refer to [About tenant IDs]({{< relref "../../configure/about-tenant-ids" >}}). +{{< /admonition >}} diff --git a/docs/sources/mimir/manage/secure/securing-alertmanager.md b/docs/sources/mimir/manage/secure/securing-alertmanager.md index 515cff1c2ab..b5867de0fbd 100644 --- a/docs/sources/mimir/manage/secure/securing-alertmanager.md +++ b/docs/sources/mimir/manage/secure/securing-alertmanager.md @@ -23,4 +23,6 @@ Although hardening the network is out of the scope for Grafana Mimir, Grafana Mi You can override the Alertmanager built-in firewall settings on a per-tenant basis in the overrides section of the [runtime configuration]({{< relref "../../configure/about-runtime-configuration" >}}). -> **Note:** You can disable the Alertmanager configuration API by setting `-alertmanager.enable-api=false`. +{{< admonition type="note" >}} +You can disable the Alertmanager configuration API by setting `-alertmanager.enable-api=false`. +{{< /admonition >}} diff --git a/docs/sources/mimir/manage/secure/securing-communications-with-tls.md b/docs/sources/mimir/manage/secure/securing-communications-with-tls.md index 452d0755d32..17738c6b0ab 100644 --- a/docs/sources/mimir/manage/secure/securing-communications-with-tls.md +++ b/docs/sources/mimir/manage/secure/securing-communications-with-tls.md @@ -19,7 +19,12 @@ components. This topic describes the process used to set up TLS. To establish secure inter-component communication in Grafana Mimir with TLS, you must generate certificates using a certificate authority (CA). The CA should be private to the organization because certificates signed by the CA will have permissions to communicate with the cluster. -> **Note**: The generated certificates are valid for 100,000 days. You can change the duration by adjusting the `-days` option in the command. We recommend that you replace the certificates every two years. +{{< admonition type="note" >}} +The generated certificates are valid for 100,000 days. +You can change the duration by adjusting the `-days` option in the command. + +You should replace certificates more regularly. +{{< /admonition >}} The following script generates self-signed certificates for the cluster. The script generates private keys `client.key`, `server.key` and certificates `client.crt`, `server.crt` for both the client and server. diff --git a/docs/sources/mimir/manage/tools/mimir-continuous-test.md b/docs/sources/mimir/manage/tools/mimir-continuous-test.md index 9a88eb302d7..5f1b6282c06 100644 --- a/docs/sources/mimir/manage/tools/mimir-continuous-test.md +++ b/docs/sources/mimir/manage/tools/mimir-continuous-test.md @@ -49,7 +49,9 @@ Mimir-continuous-test requires the endpoints of the backend Grafana Mimir cluste - `-tests.tenant-id` to the tenant ID, default to `anonymous`. - Set `-tests.smoke-test` to run the test once and immediately exit. In this mode, the process exit code is non-zero when the test fails. -> **Note:** You can run `mimir-continuous-test -help` to list all available configuration options. +{{< admonition type="note" >}} +You can run `mimir-continuous-test -help` to list all available configuration options. +{{< /admonition >}} ## How it works diff --git a/docs/sources/mimir/manage/tools/mimirtool.md b/docs/sources/mimir/manage/tools/mimirtool.md index 60d5106f29d..40df7a5371c 100644 --- a/docs/sources/mimir/manage/tools/mimirtool.md +++ b/docs/sources/mimir/manage/tools/mimirtool.md @@ -85,8 +85,13 @@ The following sections outline the commands that you can run against Grafana Mim The following commands interact with Grafana Mimir Alertmanager configuration and alert template files. -> **Note:** The Mimir Alertmanager exposes the configuration API based on the path that you set via the `-server.path-prefix` flag; it does not use the path of the `-http.alertmanager-http-prefix` flag. -> If you run Mimir with the default configuration (`-server.path-prefix`, where the default value is `/`), then only set the hostname for the `--address` flag of the `mimirtool` command; do not set a path-specific address. For example, `/` is correct, and `/alertmanager` is incorrect. +{{< admonition type="note" >}} +The Mimir Alertmanager exposes the configuration API based on the path that you set via the `-server.path-prefix` flag; it doesn't use the path of the `-http.alertmanager-http-prefix` flag. + +If you run Mimir with the default configuration, `-server.path-prefix`, where the default value is `/`, then only set the hostname for the `--address` flag of the `mimirtool` command; do not set a path-specific address. + +For example, `/` is correct, and `/alertmanager` is incorrect. +{{< /admonition >}} #### Get Alertmanager configuration @@ -329,7 +334,9 @@ The `lint` command provides YAML and PromQL expression formatting within the rul This command edits the rule file in place. To perform a trial run that does not make changes, you can use the dry run flag (`-n`). -> **Note:** This command does not verify if a query is correct and does not interact with your Grafana Mimir cluster. +{{< admonition type="note" >}} +This command doesn't verify if a query is correct and doesn't interact with your Grafana Mimir cluster. +{{< /admonition >}} ```bash mimirtool rules lint ... @@ -343,7 +350,9 @@ This `prepare` command prepares a rules file that you upload to Grafana Mimir. It lints all PromQL expressions and adds a label to your PromQL query aggregations in the file. The format of the file is the same format as shown in [rules load](#load-rule-group). -> **Note:** This command does not interact with your Grafana Mimir cluster. +{{< admonition type="note" >}} +This command doesn't interact with your Grafana Mimir cluster. +{{< /admonition >}} ```bash mimirtool rules prepare ... @@ -567,7 +576,9 @@ The `acl` command generates the label-based access control header used in Grafan The following command enables you to generate a header that you can use to enforce access control rules in Grafana Enterprise Metrics or Grafana Cloud. -> **Note**: Grafana Mimir does not support ACLs. +{{< admonition type="note" >}} +Grafana Mimir doesn't support ACLs. +{{< /admonition >}} ```bash mimirtool acl generate-header --id= --rule= @@ -716,8 +727,10 @@ The command uses the output from a previous run of `analyze grafana`, `analyze d or `analyze rule-file` to show the number of series in the Prometheus instance that are used in dashboards or rules, or both. This command also shows which metrics exist in Grafana Cloud that are _not_ in dashboards or rules. The output is a JSON file. -> **Note:** The command makes a request for every active series in the Prometheus instance. -> For Prometheus instances with a large number of active series, this command might take time to complete. +{{< admonition type="note" >}} +The command makes a request for every active series in the Prometheus instance. +For Prometheus instances with a large number of active series, this command might take time to complete. +{{< /admonition >}} ```bash mimirtool analyze prometheus --address= --id= @@ -885,7 +898,12 @@ server: http_listen_port: 80 ``` -> **Note:** As a precaution,`server.http_listen_port` is included. The default value in Grafana Mimir changed from 80 to 8080. Unless you explicitly set the port in the input configuration, the tool outputs the old default value. +{{< admonition type="note" >}} +As a precaution,the output file includes `server.http_listen_port`. +The default value in Grafana Mimir changed from 80 to 8080. + +Unless you explicitly set the port in the input configuration, the tool outputs the original default value. +{{< /admonition >}} `mimir.flags` converted output file: diff --git a/docs/sources/mimir/manage/tools/query-tee.md b/docs/sources/mimir/manage/tools/query-tee.md index c81726ab275..bfb39ddfd95 100644 --- a/docs/sources/mimir/manage/tools/query-tee.md +++ b/docs/sources/mimir/manage/tools/query-tee.md @@ -43,7 +43,9 @@ You can configure the backend endpoints by setting the `-backend.endpoints` flag For each incoming request, the query-tee clones the request and sends it to each configured backend. -> **Note:** You can configure the query-tee proxy listening ports via the `-server.http-service-port` flag for the HTTP port and `server.grpc-service-port` flag for the gRPC port. +{{< admonition type="note" >}} +You can configure the query-tee proxy listening ports via the `-server.http-service-port` flag for the HTTP port and `server.grpc-service-port` flag for the gRPC port. +{{< /admonition >}} ## How the query-tee works @@ -105,7 +107,9 @@ When a preferred backend is not configured, the query-tee uses the following alg 1. If at least one backend response status code is 2xx or 4xx, the query-tee selects the first received response whose status code is 2xx or 4xx. 1. If no backend response status code is 2xx or 4xx, the query-tee selects the first received response regardless of the status code. -> **Note:** The query-tee considers a 4xx response as a valid response to select because a 4xx status code generally means the error is caused by an invalid request and not due to a server side issue. +{{< admonition type="note" >}} +The query-tee considers a 4xx response as a valid response to select because a 4xx status code generally an invalid request and not a server side issue. +{{< /admonition >}} ### Backend results comparison @@ -117,13 +121,19 @@ The query results comparison can be enabled setting the flag `-proxy.compare-res When the query results comparison is enabled, the query-tee compares the response received from the two configured backends and logs a message for each query whose results don't match. Query-tee keeps track of the number of successful and failed comparison through the metric `cortex_querytee_responses_compared_total`. -> **Note**: Floating point sample values are compared with a tolerance that can be configured via `-proxy.value-comparison-tolerance`. The configured tolerance prevents false positives due to differences in floating point values rounding introduced by the non-deterministic series ordering within the Prometheus PromQL engine. +{{< admonition type="note" >}} +Query-tee compares Floating point sample values with a tolerance that you can configure with the `-proxy.value-comparison-tolerance` option. -> **Note**: The default value of `-proxy.compare-skip-recent-samples` is 2 minutes. This means points within results with a timestamp within 2 minutes of the current time will not be compared. -> This prevents false positives due to racing with ingestion, and, if the query selects the output of recording rules, rule evaluation. -> -> If either Mimir cluster is running with a non-default value of `-ruler.evaluation-delay-duration`, we recommend setting `-proxy.compare-skip-recent-samples` to 1 minute more than the -> value of `-ruler.evaluation-delay-duration`. +The configured tolerance prevents false positives due to differences in floating point values rounding introduced by the non-deterministic series ordering within the Prometheus PromQL engine. +{{< /admonition >}} + +{{< admonition type="note" >}} +The default value of `-proxy.compare-skip-recent-samples` is two minutes. +This means points within results with a timestamp within two minutes of the current time aren't compared. +This prevents false positives due to racing with ingestion, and, if the query selects the output of recording rules, rule evaluation. + +If either Mimir cluster is running with a non-default value of `-ruler.evaluation-delay-duration`, you should set `-proxy.compare-skip-recent-samples` to one minute more than the value of `-ruler.evaluation-delay-duration`. +{{< /admonition >}} ### Exported metrics diff --git a/docs/sources/mimir/references/architecture/components/alertmanager.md b/docs/sources/mimir/references/architecture/components/alertmanager.md index 7de7117b80a..e14b8627f7a 100644 --- a/docs/sources/mimir/references/architecture/components/alertmanager.md +++ b/docs/sources/mimir/references/architecture/components/alertmanager.md @@ -25,8 +25,13 @@ Each tenant has an isolated alert routing configuration and Alertmanager UI. Each tenant has an Alertmanager configuration that defines notifications receivers and alerting routes. The Mimir Alertmanager uses the same [configuration file](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) that the Prometheus Alertmanager uses. -> **Note:** The Mimir Alertmanager exposes the configuration API according to the path set by the `-server.path-prefix` flag. It does not use the path set by the `-http.alertmanager-http-prefix` flag. -> If you run Mimir with the default configuration (`-server.path-prefix`, where the default value is `/`), then only set the hostname for the `--address` flag of the `mimirtool` command; do not set a path-specific address. For example, `/` is correct, and `/alertmanager` is incorrect. +{{< admonition type="note" >}} +The Mimir Alertmanager exposes the configuration API according to the path set by the `-server.path-prefix` flag. +It doesn't use the path set by the `-http.alertmanager-http-prefix` flag. + +If you run Mimir with the default configuration, `-server.path-prefix`, where the default value is `/`, then only set the hostname for the `--address` flag of the `mimirtool` command; don't set a path-specific address. +For example, `/` is correct, and `/alertmanager` is incorrect. +{{< /admonition >}} You can validate a configuration file using the `mimirtool` command: @@ -66,7 +71,9 @@ When a tenant doesn't have a Alertmanager configuration, the Grafana Mimir Alert By default, there is always a fallback configuration set. You can overwrite the default fallback configuration via the `-alertmanager.configs.fallback` command-line flag. -> **Warning**: Without a fallback configuration or a tenant specific configuration, the Alertmanager UI is inaccessible and ruler notifications for that tenant fail. +{{< admonition type="warning" >}} +Without a fallback configuration or a tenant specific configuration, the Alertmanager UI is inaccessible and ruler notifications for that tenant fail. +{{< /admonition >}} ### Tenant limits @@ -87,8 +94,11 @@ You can configure the HTTP path prefix for the UI and the HTTP API: - `-http.alertmanager-http-prefix` configures the path prefix for Alertmanager endpoints. - `-alertmanager.web.external-url` configures the source URLs generated in Alertmanager alerts and from where to fetch web assets. -> **Note:** Unless you are using a reverse proxy in front of the Alertmanager API that rewrites routes, the path prefix set in `-alertmanager.web.external-url` must match the path prefix set in `-http.alertmanager-http-prefix` (`/alertmanager` by default). -> If the path prefixes do not match, HTTP requests routing might not work as expected. +{{< admonition type="note" >}} +Unless you are using a reverse proxy in front of the Alertmanager API that rewrites routes, the path prefix set in `-alertmanager.web.external-url` must match the path prefix set in `-http.alertmanager-http-prefix` which is `/alertmanager` by default. + +If the path prefixes don't match, HTTP requests routing might not work as expected. +{{< /admonition >}} ### Using a reverse proxy @@ -118,19 +128,22 @@ If the Mimir Alertmanager replica receiving the HTTP request doesn't own the ten To configure the Alertmanagers' hash ring, refer to [configuring hash rings]({{< relref "../../../configure/configure-hash-rings" >}}). -> **Note:** When running with a single tenant, scaling the number of replicas to be greater than the replication factor offers no benefits as the Mimir Alertmanager shards by tenant and not individual alerts. +{{< admonition type="note" >}} +When running with a single tenant, scaling the number of replicas to be greater than the replication factor offers no benefits as the Mimir Alertmanager shards by tenant and not individual alerts. +{{< /admonition >}} ### State The Mimir Alertmanager stores the alerts state on local disk at the location configured using `-alertmanager.storage.path`. -> **Warning:** -> When running the Mimir Alertmanager without replication, ensure persistence of the `-alertmanager.storage.path` directory to avoid losing alert state. +{{< admonition type="warning" >}} +When running the Mimir Alertmanager without replication, ensure persistence of the `-alertmanager.storage.path` directory to avoid losing alert state. +{{< /admonition >}} The Mimir Alertmanager also periodically stores the alert state in the storage backend configured with `-alertmanager-storage.backend`. When an Alertmanager starts, it attempts to load the alerts state for a given tenant from other Alertmanager replicas. If the load from other Alertmanager replicas fails, the Alertmanager falls back to the state that is periodically stored in the storage backend. -In the event of a cluster outage, this fallback mechanism recovers the backup of the previous state. Because backups are taken periodically, this fallback mechanism does not guarantee that the lastest state is restored. +In the event of a cluster outage, this fallback mechanism recovers the backup of the previous state. Because backups are taken periodically, this fallback mechanism does not guarantee that the most recent state is restored. ## Ruler configuration diff --git a/docs/sources/mimir/references/architecture/components/distributor.md b/docs/sources/mimir/references/architecture/components/distributor.md index c7f9ca4ecac..36f04ca7fb8 100644 --- a/docs/sources/mimir/references/architecture/components/distributor.md +++ b/docs/sources/mimir/references/architecture/components/distributor.md @@ -35,7 +35,9 @@ The distributor validation includes the following checks: - Each exemplar has a timestamp and at least one non-empty label name and value pair. - Each exemplar has no more than 128 labels. -> **Note:** For each tenant, you can override the validation checks by modifying the overrides section of the [runtime configuration]({{< relref "../../../configure/about-runtime-configuration" >}}). +{{< admonition type="note" >}} +For each tenant, you can override the validation checks by modifying the overrides section of the [runtime configuration]({{< relref "../../../configure/about-runtime-configuration" >}}). +{{< /admonition >}} ## Rate limiting @@ -61,9 +63,13 @@ Use the following flags to configure the rate limits: - `-distributor.ingestion-rate-limit`: Ingestion rate limit, which is per tenant, and which is in samples per second - `-distributor.ingestion-burst-size`: Ingestion burst size (in number of samples) allowed, which is per tenant -> **Note:** You can override rate limiting on a per-tenant basis by setting `request_rate`, `ingestion_rate`, `request_burst_size` and `ingestion_burst_size` in the overrides section of the runtime configuration. +{{< admonition type="note" >}} +You can override rate limiting on a per-tenant basis by setting `request_rate`, `ingestion_rate`, `request_burst_size` and `ingestion_burst_size` in the overrides section of the runtime configuration. +{{< /admonition >}} -> **Note:** By default, Prometheus remote write doesn't retry requests on 429 HTTP response status code. To modify this behavior, use `retry_on_http_429: true` in the Prometheus [`remote_write` configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). +{{< admonition type="note" >}} +By default, Prometheus remote write doesn't retry requests on 429 HTTP response status code. To modify this behavior, use `retry_on_http_429: true` in the Prometheus [`remote_write` configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). +{{< /admonition >}} ### Configuration @@ -106,10 +112,14 @@ The distributor waits for a successful response from `n`/2 + 1 ingesters, where We recommend randomly load balancing write requests across distributor instances. If you're running Grafana Mimir in a Kubernetes cluster, you can define a Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/) as ingress for the distributors. -> **Note:** A Kubernetes Service balances TCP connections across Kubernetes endpoints and does not balance HTTP requests within a single TCP connection. -> If you enable HTTP persistent connections (HTTP keep-alive), because Prometheus uses HTTP keep-alive, it re-uses the same TCP connection for each remote-write HTTP request of a remote-write shard. -> This can cause distributors to receive an uneven distribution of remote-write HTTP requests. -> To improve the balancing of requests between distributors, consider increasing `min_shards` in the Prometheus [remote write config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). +{{< admonition type="note" >}} +A Kubernetes Service balances TCP connections across Kubernetes endpoints and doesn't balance HTTP requests within a single TCP connection. + +If you enable HTTP persistent connections, also known as HTTP keep-alive, Prometheus reuses the same TCP connection for each remote-write HTTP request of a remote-write shard. +This can cause distributors to receive an uneven distribution of remote-write HTTP requests. + +To improve the balancing of requests between distributors, consider increasing `min_shards` in the Prometheus [`remote write` configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). +{{< /admonition >}} ## Configuration diff --git a/docs/sources/mimir/references/architecture/components/ingester.md b/docs/sources/mimir/references/architecture/components/ingester.md index bc98c4c3477..a42da99b275 100644 --- a/docs/sources/mimir/references/architecture/components/ingester.md +++ b/docs/sources/mimir/references/architecture/components/ingester.md @@ -56,7 +56,9 @@ Writes to the Mimir cluster are successful if a majority of ingesters received t If the Mimir cluster loses a minority of ingesters, the in-memory series samples held by the lost ingesters are available in at least one other ingester, meaning no time series samples are lost. If a majority of ingesters fail, time series might be lost if the failure affects all the ingesters holding the replicas of a specific time series. -> **Note:** Replication only happens at write time. If an ingester is unavailable during a period when writes are actively being written to other ingesters, that particular ingester will never recover those missed samples. +{{< admonition type="note" >}} +Replication only happens at write time. If an ingester is unavailable during a period when writes are actively being written to other ingesters, that particular ingester will never recover those missed samples. +{{< /admonition >}} ### Write-ahead log diff --git a/docs/sources/mimir/references/architecture/components/overrides-exporter.md b/docs/sources/mimir/references/architecture/components/overrides-exporter.md index 93a303392c5..f0ddb3b3cc4 100644 --- a/docs/sources/mimir/references/architecture/components/overrides-exporter.md +++ b/docs/sources/mimir/references/architecture/components/overrides-exporter.md @@ -21,9 +21,10 @@ For more information about configuring overrides, refer to [Runtime configuratio The overrides-exporter must be explicitly enabled. -> **Warning:** -> The metrics emitted by the overrides-exporter have high cardinality. -> It's recommended to run only a single replica of the overrides-exporter to limit that cardinality. +{{< admonition type="warning" >}} +The metrics emitted by the overrides-exporter have high cardinality. +It's recommended to run only a single replica of the overrides-exporter to limit that cardinality. +{{< /admonition >}} With a `runtime.yaml` file as follows: diff --git a/docs/sources/mimir/references/architecture/components/querier.md b/docs/sources/mimir/references/architecture/components/querier.md index df88532ac4a..bc703eddc96 100644 --- a/docs/sources/mimir/references/architecture/components/querier.md +++ b/docs/sources/mimir/references/architecture/components/querier.md @@ -82,11 +82,17 @@ Using the metadata cache reduces the number of API calls to long-term storage an To enable the metadata cache, set `-blocks-storage.bucket-store.metadata-cache.backend`. -> **Note**: Currently, only the `memcached` backend is supported. The Memcached client includes additional configuration available via flags that begin with the prefix `-blocks-storage.bucket-store.metadata-cache.memcached.*`. +{{< admonition type="note" >}} +Currently, Mimir supports caching with the `memcached` backend. + +The Memcached client includes additional configuration available via flags that begin with the prefix `-blocks-storage.bucket-store.metadata-cache.memcached.*`. +{{< /admonition >}} Additional flags for configuring the metadata cache begin with the prefix `-blocks-storage.bucket-store.metadata-cache.*`. By configuring the TTL to zero or a negative value, caching of given item type is disabled. -> **Note:** The same Memcached backend cluster should be shared between store-gateways and queriers. +{{< admonition type="note" >}} +You should use the same Memcached backend cluster for both the store-gateways and queriers. +{{< /admonition >}} ## Querier configuration diff --git a/docs/sources/mimir/references/architecture/components/query-frontend/index.md b/docs/sources/mimir/references/architecture/components/query-frontend/index.md index feff82f55ac..76ab47efb30 100644 --- a/docs/sources/mimir/references/architecture/components/query-frontend/index.md +++ b/docs/sources/mimir/references/architecture/components/query-frontend/index.md @@ -26,7 +26,9 @@ We recommend that you run at least two query-frontend replicas for high-availabi The following steps describe how a query moves through the query-frontend. -> **Note:** In this scenario, a query-scheduler is not deployed. +{{< admonition type="note" >}} +This scenario doesn't deploy a query-scheduler. +{{< /admonition >}} 1. A query-frontend receives a query. 1. If the query is a range query, the query-frontend [splits it by time]({{< relref "#splitting" >}}) into multiple smaller queries that can be parallelized. diff --git a/docs/sources/mimir/references/architecture/components/query-scheduler/index.md b/docs/sources/mimir/references/architecture/components/query-scheduler/index.md index f2fa504d482..5083f53c4cf 100644 --- a/docs/sources/mimir/references/architecture/components/query-scheduler/index.md +++ b/docs/sources/mimir/references/architecture/components/query-scheduler/index.md @@ -48,9 +48,16 @@ To use the query-scheduler with DNS-based service discovery, configure the query - Query-frontend: `-query-frontend.scheduler-address` - Querier: `-querier.scheduler-address` -> **Note**: The configured query-scheduler address should be in the `host:port` format. If multiple query-schedulers are running, the host should be a DNS resolving to all query-scheduler instances. +{{< admonition type="note" >}} +The configured query-scheduler address should be in the `host:port` format. -> **Note:** The querier pulls queries only from the query-frontend or the query-scheduler, but not both. `-querier.frontend-address` and `-querier.scheduler-address` options are mutually exclusive, and only one option can be set. +If multiple query-schedulers are running, the host should be a DNS name resolving to all query-scheduler instances. +{{< /admonition >}} + +{{< admonition type="note" >}} +The querier pulls queries only from the query-frontend or the query-scheduler, but not both. +`-querier.frontend-address` and `-querier.scheduler-address` options are mutually exclusive and you can only set one option. +{{< /admonition >}} ### Ring-based service discovery @@ -104,7 +111,9 @@ To migrate the query-scheduler from [DNS-based service discovery](#dns-based-ser # - Ecd: -query-scheduler.ring.etcd.* ``` -> **Note:** If your Mimir cluster is deployed using Jsonnet, see [Migrate query-scheduler from DNS-based to ring-based service discovery]({{< relref "../../../../set-up/jsonnet/migrate-query-scheduler-from-dns-to-ring-based-service-discovery" >}}). +{{< admonition type="note" >}} +If you deploy your Mimir cluster with Jsonnet, refer to [Migrate query-scheduler from DNS-based to ring-based service discovery]({{< relref "../../../../set-up/jsonnet/migrate-query-scheduler-from-dns-to-ring-based-service-discovery" >}}). +{{< /admonition >}} ## Operational considerations diff --git a/docs/sources/mimir/references/architecture/components/ruler/index.md b/docs/sources/mimir/references/architecture/components/ruler/index.md index 94944fb61d4..0f067e0a959 100644 --- a/docs/sources/mimir/references/architecture/components/ruler/index.md +++ b/docs/sources/mimir/references/architecture/components/ruler/index.md @@ -28,7 +28,9 @@ Configuration of the built-in querier and distributor uses their respective conf - [Querier]({{< relref "../../../../configure/configuration-parameters#querier" >}}) - [Distributor]({{< relref "../../../../configure/configuration-parameters#distributor" >}}) -> **Note**: When this mode is used, no query acceleration techniques are used and the evaluation of very high cardinality queries could take longer than the evaluation interval, eventually leading to missing data points in the evaluated recording rules. +{{< admonition type="note" >}} +When you use the internal mode, the ruler uses no query acceleration techniques and the evaluation of very high cardinality queries could take longer than the evaluation interval, which may lead to missing data points in the evaluated recording rules. +{{< /admonition >}} [//]: # "Diagram source of ruler interactions https://docs.google.com/presentation/d/1LemaTVqa4Lf_tpql060vVoDGXrthp-Pie_SQL7qwHjc/edit#slide=id.g11658e7e4c6_0_938" @@ -94,12 +96,15 @@ unavailability). The time series used during evaluation of federated rules will have the `__tenant_id__` label, similar to how it is present on series returned with cross-tenant query federation. -> **Note**: Federated rule groups allow data from multiple source tenants to be written into a single -> destination tenant. This makes the existing separation of tenants' data less clear. For example, `tenant-a` has a -> federated rule group that aggregates over `tenant-b`'s data (e.g. `sum(metric_b)`) and writes the result back -> into `tenant-a`'s storage (e.g. as metric `sum:metric_b`). Now part of `tenant-b`'s data is copied to `tenant-a` (albeit -> aggregated). Have this in mind when configuring the access control layer in front of mimir and when enabling federated -> rules via `-ruler.tenant-federation.enabled`. +{{< admonition type="note" >}} +Federated rule groups allow data from multiple source tenants to be written into a single destination tenant. +This makes the separation of tenants' data less clear. + +For example, `tenant-a` has a federated rule group that aggregates over `tenant-b`'s data like `sum(metric_b)` and writes the result back into `tenant-a`'s storage as the metric `sum:metric_b`. +Now `tenant-a` contains some of `tenant-b`'s data. + +Have this in mind when configuring the access control layer in front of Mimir and when enabling federated rules via `-ruler.tenant-federation.enabled`. +{{< /admonition >}} ## Sharding @@ -143,7 +148,9 @@ The ruler supports the following backends: The `local` storage backend reads [Prometheus recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) from the local filesystem. -> **Note:** Local storage is a read-only backend that does not support the creation and deletion of rules through the [Configuration API]({{< relref "#via-the-http-configuration-api" >}}). +{{< admonition type="note" >}} +Local storage is a read-only backend that doesn't support the creation and deletion of rules through the [Configuration API](#via-the-http-configuration-api). +{{< /admonition >}} When all rulers have the same rule files, local storage supports ruler sharding. To facilitate sharding in Kubernetes, mount a [Kubernetes ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) into every ruler pod. diff --git a/docs/sources/mimir/references/architecture/components/store-gateway.md b/docs/sources/mimir/references/architecture/components/store-gateway.md index 28b272b88e4..56e6400118a 100644 --- a/docs/sources/mimir/references/architecture/components/store-gateway.md +++ b/docs/sources/mimir/references/architecture/components/store-gateway.md @@ -66,13 +66,19 @@ When the querier queries blocks via a store-gateway, the response contains the l If a querier attempts to query a block that the store-gateway has not loaded, the querier retries the query on a different store-gateway up to the `-store-gateway.sharding-ring.replication-factor` value, which by default is `3`. The query fails if the block can't be successfully queried from any replica. -> **Note:** You must configure the [hash ring]({{< relref "../hash-ring" >}}) via the `-store-gateway.sharding-ring.*` flags or their respective YAML configuration parameters. +{{< admonition type="note" >}} +You must configure the [hash ring]({{< relref "../hash-ring" >}}) via the `-store-gateway.sharding-ring.*` flags or their respective YAML configuration parameters. +{{< /admonition >}} ### Sharding strategy The store-gateway uses shuffle-sharding to divide the blocks of each tenant across a subset of store-gateway instances. -> **Note:** When shuffle-sharding is in use, the number of store-gateway instances that load the blocks of a tenant is limited, which means that the blast radius of issues introduced by the tenant's workload is confined to its shard instances. +{{< admonition type="note" >}} +When shuffle-sharding is in use, only a subset of store-gateway instances load the blocks of a tenant. + +This confines blast radius of issues introduced by the tenant's workload to its shard instances. +{{< /admonition >}} The `-store-gateway.tenant-shard-size` flag (or their respective YAML configuration parameters) determines the default number of store-gateway instances per tenant. The `store_gateway_tenant_shard_size` in the limits overrides can override the shard size on a per-tenant basis. @@ -188,7 +194,9 @@ Chunks can only be cached in Memcached. To enable chunks cache, set `-blocks-storage.bucket-store.chunks-cache.backend=memcached`. You can configure the Memcached client via flags that include the prefix `-blocks-storage.bucket-store.chunks-cache.memcached.*`. -> **Note:** There are additional low-level flags that begin with the prefix `-blocks-storage.bucket-store.chunks-cache.*` that you can use to configure chunks cache. +{{< admonition type="note" >}} +There are additional low-level flags that begin with the prefix `-blocks-storage.bucket-store.chunks-cache.*` that you can use to configure chunks cache. +{{< /admonition >}} ### Metadata cache @@ -204,11 +212,17 @@ Using the metadata cache reduces the number of API calls to long-term storage an To enable metadata cache, set `-blocks-storage.bucket-store.metadata-cache.backend`. -> **Note**: Currently, only `memcached` backend is supported. The Memcached client includes additional configuration available via flags that begin with the prefix `-blocks-storage.bucket-store.metadata-cache.memcached.*`. +{{< admonition type="note" >}} +Mimir only supports the `memcached` backend for the metadata cache. + +The Memcached client includes additional configuration available via flags that begin with the prefix `-blocks-storage.bucket-store.metadata-cache.memcached.*`. +{{< /admonition >}} Additional flags for configuring metadata cache begin with the prefix `-blocks-storage.bucket-store.metadata-cache.*`. By configuring TTL to zero or a negative value, caching of given item type is disabled. -> **Note:** The same memcached backend cluster should be shared between store-gateways and queriers.\_ +{{< admonition type="note" >}} +You should use the same Memcached backend cluster for both the store-gateways and queriers. +{{< /admonition >}} ## Store-gateway HTTP endpoints diff --git a/docs/sources/mimir/references/architecture/deployment-modes/index.md b/docs/sources/mimir/references/architecture/deployment-modes/index.md index 4be9936ef5d..5b7ed7d1467 100644 --- a/docs/sources/mimir/references/architecture/deployment-modes/index.md +++ b/docs/sources/mimir/references/architecture/deployment-modes/index.md @@ -50,8 +50,9 @@ If you are interested in deploying Grafana Mimir in microservices mode, we recom ## Read-Write mode -> **Warning:** -> Read-Write deployment mode is experimental. +{{< admonition type="warning" >}} +Read-Write deployment mode is experimental. +{{< /admonition >}} The read-write mode provides an alternative to monolithic and microservices modes. diff --git a/docs/sources/mimir/references/architecture/key-value-store.md b/docs/sources/mimir/references/architecture/key-value-store.md index 871cefae7b1..e6c09d78f04 100644 --- a/docs/sources/mimir/references/architecture/key-value-store.md +++ b/docs/sources/mimir/references/architecture/key-value-store.md @@ -32,7 +32,9 @@ We recommend that you use memberlist to run Grafana Mimir. To configure memberlist, refer to [configuring hash rings]({{< relref "../../configure/configure-hash-rings" >}}). -> **Note:** The Gossip-based memberlist protocol is not supported for the [optional distributor high-availability tracker]({{< relref "../../configure/configure-high-availability-deduplication" >}}). +{{< admonition type="note" >}} +The Gossip-based memberlist protocol isn't supported for the [optional distributor high-availability tracker]({{< relref "../../configure/configure-high-availability-deduplication" >}}). +{{< /admonition >}} ### Consul diff --git a/docs/sources/mimir/references/http-api/index.md b/docs/sources/mimir/references/http-api/index.md index 444c8b6e919..b533aa02e6d 100644 --- a/docs/sources/mimir/references/http-api/index.md +++ b/docs/sources/mimir/references/http-api/index.md @@ -141,7 +141,9 @@ GET /config This endpoint displays the configuration currently applied to Grafana Mimir including default values and settings via CLI flags. This endpoint provides the configuration in YAML format and masks sensitive data. -> **Note**: The exported configuration doesn't include the per-tenant overrides. +{{< admonition type="note" >}} +The exported configuration doesn't include the per-tenant overrides. +{{< /admonition >}} #### Different modes @@ -347,7 +349,9 @@ GET /distributor/all_user_stats This endpoint displays a web page that shows per-tenant statistics updated in real time, including the total number of active series across all ingesters and the current ingestion rate displayed in samples per second. -> **Note:** This endpoint requires all ingesters to be `ACTIVE` in the ring for a successful response. +{{< admonition type="note" >}} +This endpoint requires all ingesters to be `ACTIVE` in the ring for a successful response. +{{< /admonition >}} ### HA tracker status @@ -376,7 +380,9 @@ If no tenant is specified, all tenants are flushed. The flush endpoint also accepts a `wait=true` parameter, which makes the call synchronous, and only returns a status code after flushing completes. -> **Note**: The returned status code does not reflect the result of flush operation. +{{< admonition type="note" >}} +The returned status code doesn't reflect the result of flush operation. +{{< /admonition >}} ### Prepare for Shutdown @@ -759,7 +765,9 @@ This endpoint can be disabled via the `-ruler.enable-api` CLI flag (or its respe Requires [authentication](#authentication). -> **Note:** To list all rule groups from Mimir, use [`mimirtool rules list` command]({{< relref "../../manage/tools/mimirtool#list-rules" >}}). +{{< admonition type="note" >}} +To list all rule groups from Mimir, use [`mimirtool rules list` command]({{< relref "../../manage/tools/mimirtool#list-rules" >}}). +{{< /admonition >}} **Example response** @@ -854,7 +862,9 @@ This endpoint can be disabled via the `-ruler.enable-api` CLI flag (or its respe Requires [authentication](#authentication). -> **Note:** To retrieve a single rule group from Mimir, use [`mimirtool rules get` command]({{< relref "../../manage/tools/mimirtool#get-rule-group" >}}) . +{{< admonition type="note" >}} +To retrieve a single rule group from Mimir, use [`mimirtool rules get` command]({{< relref "../../manage/tools/mimirtool#get-rule-group" >}}) . +{{< /admonition >}} ### Set rule group @@ -870,10 +880,15 @@ This endpoint can be disabled via the `-ruler.enable-api` CLI flag (or its respe Requires [authentication](#authentication). -> **Note:** To load one or more rule groups into Mimir, use [`mimirtool rules load` command]({{< relref "../../manage/tools/mimirtool#load-rule-group" >}}) . +{{< admonition type="note" >}} +To load one or more rule groups into Mimir, use [`mimirtool rules load` command]({{< relref "../../manage/tools/mimirtool#load-rule-group" >}}) . +{{< /admonition >}} -> **Note:** When using `curl` send the request body from a file, ensure that you use the `--data-binary` flag instead of `-d`, `--data`, or `--data-ascii`. -> The latter options do not preserve carriage returns and newlines. +{{< admonition type="note" >}} +When using `curl` to send the request body from a file, ensure that you use the `--data-binary` flag instead of `-d`, `--data`, or `--data-ascii`. + +The latter options don't preserve carriage returns and newlines. +{{< /admonition >}} #### Example request body @@ -898,7 +913,9 @@ This endpoint can be disabled via the `-ruler.enable-api` CLI flag (or its respe Requires [authentication](#authentication). -> **Note:** To delete a rule group from Mimir, use [`mimirtool rules delete` command]({{< relref "../../manage/tools/mimirtool#delete-rule-group" >}}). +{{< admonition type="note" >}} +To delete a rule group from Mimir, use [`mimirtool rules delete` command]({{< relref "../../manage/tools/mimirtool#delete-rule-group" >}}). +{{< /admonition >}} ### Delete namespace @@ -986,7 +1003,9 @@ This endpoint can be enabled and disabled via the `-alertmanager.enable-api` CLI Requires [authentication](#authentication). -> **Note:** To retrieve a tenant's Alertmanager configuration from Mimir, use [`mimirtool alertmanager get` command]({{< relref "../../manage/tools/mimirtool#get-alertmanager-configuration" >}}). +{{< admonition type="note" >}} +To retrieve a tenant's Alertmanager configuration from Mimir, use [`mimirtool alertmanager get` command]({{< relref "../../manage/tools/mimirtool#get-alertmanager-configuration" >}}). +{{< /admonition >}} ### Set Alertmanager configuration @@ -1004,10 +1023,15 @@ This endpoint can be enabled and disabled via the `-alertmanager.enable-api` CLI Requires [authentication](#authentication). -> **Note:** To load a tenant's Alertmanager configuration to Mimir, use [`mimirtool alertmanager load` command]({{< relref "../../manage/tools/mimirtool#load-alertmanager-configuration" >}}). +{{< admonition type="note" >}} +To load a tenant's Alertmanager configuration to Mimir, use [`mimirtool alertmanager load` command]({{< relref "../../manage/tools/mimirtool#load-alertmanager-configuration" >}}). +{{< /admonition >}} + +{{< admonition type="note" >}} +When using `curl` to send the request body from a file, ensure that you use the `--data-binary` flag instead of `-d`, `--data`, or `--data-ascii`. -> **Note:** When using `curl` send the request body from a file, ensure that you use the `--data-binary` flag instead of `-d`, `--data`, or `--data-ascii`. -> The latter options do not preserve carriage returns and newlines. +The latter options don't preserve carriage returns and newlines. +{{< /admonition >}} #### Example request body @@ -1044,7 +1068,9 @@ This endpoint can be enabled and disabled via the `-alertmanager.enable-api` CLI Requires [authentication](#authentication). -> **Note:** To delete a tenant's Alertmanager configuration from Mimir, use [`mimirtool alertmanager delete` command]({{< relref "../../manage/tools/mimirtool#delete-alertmanager-configuration" >}}). +{{< admonition type="note" >}} +To delete a tenant's Alertmanager configuration from Mimir, use [`mimirtool alertmanager delete` command]({{< relref "../../manage/tools/mimirtool#delete-alertmanager-configuration" >}}). +{{< /admonition >}} ## Store-gateway diff --git a/docs/sources/mimir/release-notes/v2.10.md b/docs/sources/mimir/release-notes/v2.10.md index ff906c3e2df..ac3f2e39e03 100644 --- a/docs/sources/mimir/release-notes/v2.10.md +++ b/docs/sources/mimir/release-notes/v2.10.md @@ -121,7 +121,9 @@ The following configuration option defaults were changed: ### 2.10.2 -> **Note**: This release contains a known performance regression fixed in 2.10.3 +{{< admonition type="note" >}} +This release contains a known performance regression fixed in 2.10.3 +{{< /admonition >}} - [BUGFIX] Update grpc-go library to 1.57.1 and `golang.org/x/net` to `0.17`, which include fix for CVE-2023-44487. [PR 6349](https://github.com/grafana/mimir/pull/6349) diff --git a/docs/sources/mimir/set-up/jsonnet/configure-autoscaling.md b/docs/sources/mimir/set-up/jsonnet/configure-autoscaling.md index dcc18963348..1585dfb02b4 100644 --- a/docs/sources/mimir/set-up/jsonnet/configure-autoscaling.md +++ b/docs/sources/mimir/set-up/jsonnet/configure-autoscaling.md @@ -17,7 +17,7 @@ Mimir Jsonnet supports autoscaling for the following components: - [Querier]({{< relref "../../references/architecture/components/querier" >}}) - [Distributor]({{< relref "../../references/architecture/components/distributor" >}}) -Autoscaling, which is based on Prometheus metrics and [KEDA (Kubernetes-based Event Driven Autoscaler)](https://keda.sh), uses Kubernetes’ Horizontal Pod Autoscaler (HPA). +Autoscaling, which is based on Prometheus metrics and [Kubernetes-based Event Driven Autoscaler (KEDA)](https://keda.sh), uses Kubernetes’ Horizontal Pod Autoscaler (HPA). HPA is not configured directly in Jsonnet but it's created and updated by KEDA. KEDA is an operator, running in the Kubernetes cluster, which is responsible to simplify the setup of HPA with custom metrics (Prometheus in our case). @@ -80,7 +80,11 @@ mimir { } ``` -> **Note**: KEDA will not be installed by Mimir jsonnet. You can follow the [Deploying KEDA](https://keda.sh/docs/latest/deploy/) instructions to install it in your Kubernetes cluster. +{{< admonition type="note" >}} +The Mimir Jsonet doesn't install KEDA. + +To install KEDA, follow the instructions in [Deploying KEDA](https://keda.sh/docs/latest/deploy/). +{{< /admonition >}} ## How to disable autoscaling diff --git a/docs/sources/mimir/set-up/jsonnet/configure-deployment-mode.md b/docs/sources/mimir/set-up/jsonnet/configure-deployment-mode.md index b19ff8a73f6..a59b9728b13 100644 --- a/docs/sources/mimir/set-up/jsonnet/configure-deployment-mode.md +++ b/docs/sources/mimir/set-up/jsonnet/configure-deployment-mode.md @@ -13,8 +13,9 @@ Grafana Mimir supports multiple [deployment modes]({{< relref "../../references/ ## Use Read-Write deployment mode -> **Warning:** -> Read-Write deployment mode is currently considered experimental. +{{< admonition type="warning" >}} +Read-Write deployment mode is experimental. +{{< /admonition >}} ### Requirements diff --git a/docs/sources/mimir/set-up/jsonnet/configure-ruler.md b/docs/sources/mimir/set-up/jsonnet/configure-ruler.md index d76349c8f92..b569e48ea9d 100644 --- a/docs/sources/mimir/set-up/jsonnet/configure-ruler.md +++ b/docs/sources/mimir/set-up/jsonnet/configure-ruler.md @@ -44,8 +44,10 @@ To get started, use the `local` client type for initial testing: If you are using object storage, you must set `ruler_storage_bucket_name` to the name of the bucket that you want to use. -> **Note:** If ruler object storage credentials differ from the ones defined in the common section, you need to manually provide them by using additional command line arguments. -> For more information, see [Grafana Mimir configuration parameters: ruler_storage]({{< relref "../../configure/configuration-parameters#ruler_storage" >}}). +{{< admonition type="note" >}} +If ruler object storage credentials differ from the ones defined in the common section, you need to manually provide them by using additional command line arguments. +For more information, refer to [Grafana Mimir configuration parameters: ruler_storage]({{< relref "../../configure/configuration-parameters#ruler_storage" >}}). +{{< /admonition >}} ## Operational modes @@ -56,17 +58,22 @@ To enable the remote operational mode, add the following code to the Jsonnet: ```jsonnet { - _config+:: { + _config+:: {n ruler_remote_evaluation_enabled: true, }, } ``` -> **Note:** To support the _remote_ operational mode, a separate query path is deployed to evaluate rules that consist of three additional Kubernetes deployments: -> -> - `ruler-query-frontend` -> - `ruler-query-scheduler` -> - `ruler-querier` +{{< admonition type="note" >}} +To support the _remote_ operational mode, the configuration includes three additional Kubernetes Deployments as a separate query path. + +These are: + +- `ruler-query-frontend` +- `ruler-query-scheduler` +- `ruler-querier` + +{{< /admonition >}} ### Migrate to remote evaluation diff --git a/docs/sources/mimir/set-up/jsonnet/deploy.md b/docs/sources/mimir/set-up/jsonnet/deploy.md index 97b57cca56c..063b52dc0a4 100644 --- a/docs/sources/mimir/set-up/jsonnet/deploy.md +++ b/docs/sources/mimir/set-up/jsonnet/deploy.md @@ -101,6 +101,13 @@ You can use [Tanka](https://tanka.dev/) and [jsonnet-bundler](https://github.com kubectl apply -k manifests/ ``` - > **Note**: The generated Kubernetes manifests create resources in the `default` namespace. To use a different namespace, change the `namespace` configuration option in the `environments/default/main.jsonnet` file, and re-generate the Kubernetes manifests. - - > **Note:** A Jsonnet-based installation uses etcd for the HA tracker by default. The Jsonnet-based installation creates the etcd using the [etcd-operator](https://github.com/coreos/etcd-operator). Before applying the `tk apply` or `kubectl apply` command , make sure that you have the etcd-operator running in your Kubernetes cluster. + {{< admonition type="note" >}} + The generated Kubernetes manifests create resources in the `default` namespace. + To use a different namespace, change the `namespace` configuration option in the `environments/default/main.jsonnet` file, and re-generate the Kubernetes manifests. + {{< /admonition >}} + + {{< admonition type="warning" >}} + A Jsonnet-based installation uses etcd for the HA tracker by default. + The Jsonnet-based installation creates the etcd using the [etcd-operator](https://github.com/coreos/etcd-operator). + Before applying the `tk apply` or `kubectl apply` command, make sure that you have the etcd-operator running in your Kubernetes cluster. + {{< /admonition >}} diff --git a/docs/sources/mimir/set-up/jsonnet/migrate-from-consul-to-memberlist.md b/docs/sources/mimir/set-up/jsonnet/migrate-from-consul-to-memberlist.md index b0451a39d52..13b3ffb0963 100644 --- a/docs/sources/mimir/set-up/jsonnet/migrate-from-consul-to-memberlist.md +++ b/docs/sources/mimir/set-up/jsonnet/migrate-from-consul-to-memberlist.md @@ -121,7 +121,11 @@ We use `multikv_migration_teardown` to preserve runtime configuration for `multi All `cortex_multikv_*` metrics are only exposed by components that use `multi` KV store. As components restart, these metrics will disappear. -> **Note**: setting `multikv_migration_enabled: false` while keeping `memberlist_ring_enabled: true` will also remove Consul! That's expected, since Consul is not used anymore – mirroring to it was disabled in step 4. +{{< admonition type="note" >}} +Setting `multikv_migration_enabled: false` while keeping `memberlist_ring_enabled: true` removes the Consul workload. + +That's expected, since Consul isn't used anymore—you disabled mirroring to it in step 4. +{{< /admonition >}} If you need to keep consul running, you can explicitly set `consul_enabled: true` in `_config`. diff --git a/docs/sources/mimir/set-up/jsonnet/migrate-from-microservices-to-read-write.md b/docs/sources/mimir/set-up/jsonnet/migrate-from-microservices-to-read-write.md index 19b740c1dfc..7532142ecaf 100644 --- a/docs/sources/mimir/set-up/jsonnet/migrate-from-microservices-to-read-write.md +++ b/docs/sources/mimir/set-up/jsonnet/migrate-from-microservices-to-read-write.md @@ -11,8 +11,9 @@ weight: 40 # Migrate from microservices to read-write mode without downtime -> **Warning:** -> Read-Write mode is considered experimental, as well as migrating between the modes. +{{< admonition type="warning" >}} +Read-Write mode, and migrating between modes, is experimental. +{{< /admonition >}} At a high level, the steps involved are as follows: @@ -293,10 +294,11 @@ Wait the next TSDB head compaction for ingesters (2 hours). ### Step 7.2: Scale down ingesters -> **Warning:** -> You must follow the shutdown ingester procedure to avoid data loss. +{{< admonition type="warning" >}} +You must follow the shutdown ingester procedure to avoid data loss. +{{< /admonition >}} -Follow the procedure for [shutting down ingesters](< relref "../../run-production-environment/scaling-out.md#scaling-down-ingesters-deployed-in-multiple-zones" >}}) in `ingester-zone-a`. +Follow the procedure for [shutting down ingesters]({{< relref "../../manage/run-production-environment/scaling-out.md#scaling-down-ingesters-deployed-in-multiple-zones" >}}) in `ingester-zone-a`. Scale down zone-a replicas (this can be done before waiting for step 4 in the shutdown procedure): diff --git a/docs/sources/mimir/set-up/migrate/migrate-from-cortex.md b/docs/sources/mimir/set-up/migrate/migrate-from-cortex.md index 78d49accf31..194d622ef40 100644 --- a/docs/sources/mimir/set-up/migrate/migrate-from-cortex.md +++ b/docs/sources/mimir/set-up/migrate/migrate-from-cortex.md @@ -15,7 +15,11 @@ weight: 10 As an operator, you can migrate a Jsonnet deployment of [Cortex](https://cortexmetrics.io/) to Grafana Mimir. The overview includes the steps required for any environment. To migrate deployment environments with Jsonnet, see [Migrate to Grafana Mimir using Jsonnet]({{< relref "#migrate-to-grafana-mimir-using-jsonnet" >}}). -> **Note:** This document was tested with Cortex versions 1.10 and 1.11. It might work with more recent versions of Cortex, but that is not guaranteed. +{{< admonition type="warning" >}} +This document was tested with Cortex versions 1.10 and 1.11. + +It might work with more recent versions of Cortex, but it's not guaranteed. +{{< /admonition >}} To migrate a Helm deployment of Cortex refer to [Migrate from Cortex](/docs/helm-charts/mimir-distributed/latest/migration-guides/migrate-from-cortex). @@ -46,7 +50,9 @@ It provides a simple migration by generating Mimir configuration from Cortex con ## Notable changes -> **Note:** For the full list of changes, refer to Mimir’s [CHANGELOG](https://github.com/grafana/mimir/blob/main/CHANGELOG.md). +{{< admonition type="note" >}} +For the full list of changes, refer to Mimir’s [CHANGELOG](https://github.com/grafana/mimir/blob/main/CHANGELOG.md). +{{< /admonition >}} - The Grafana Mimir HTTP server defaults to listening on port 8080; Cortex defaults to listening on port 80. To maintain port 80 as the listening port, set `-server.http-listen-port=80`. @@ -161,8 +167,10 @@ jb install github.com/grafana/mimir/operations/mimir-mixin@main a. Add the dashboards to Grafana. The dashboards replace your Cortex dashboards and continue to work for monitoring Cortex deployments. - > **Note:** Resource dashboards are now enabled by default and require additional metrics sources. - > To understand the required metrics sources, refer to [Additional resources metrics]({{< relref "../../manage/monitor-grafana-mimir/requirements#additional-resources-metrics" >}}). + {{< admonition type="note" >}} + Resource dashboards are enabled by default and require additional metrics sources. + To understand the required metrics sources, refer to [Additional resources metrics]({{< relref "../../manage/monitor-grafana-mimir/requirements#additional-resources-metrics" >}}). + {{< /admonition >}} b. Install the recording and alerting rules into the ruler or a Prometheus server. diff --git a/docs/sources/mimir/set-up/migrate/migrate-from-thanos-or-prometheus.md b/docs/sources/mimir/set-up/migrate/migrate-from-thanos-or-prometheus.md index c4943889c29..1134a4003ef 100644 --- a/docs/sources/mimir/set-up/migrate/migrate-from-thanos-or-prometheus.md +++ b/docs/sources/mimir/set-up/migrate/migrate-from-thanos-or-prometheus.md @@ -46,8 +46,9 @@ Use Grafana mimirtool to upload each block, such as those identified by the prev mimirtool backfill --address=http:// --id= ... ``` -> **Note**: If you need to authenticate against Grafana Mimir, you can provide an API key via the `--key` flag, -> for example `--key=$(cat token.txt)`. +{{< admonition type="note" >}} +If you need to authenticate against Grafana Mimir, you can provide an API key via the `--key` flag, for example `--key=$(cat token.txt)`. +{{< /admonition >}} Grafana Mimir performs some sanitization and validation of each block's metadata. As a result, it rejects Thanos blocks due to unsupported labels. @@ -63,7 +64,10 @@ In the Grafana Mimir 2.1 (or earlier) release, the ingesters added an external l In the Grafana Mimir 2.2 (or later) release, blocks no longer have a label that identifies the tenant. -> **Note**: Blocks from Prometheus do not have any external labels stored in them; only blocks from Thanos use labels. +{{< admonition type="note" >}} +Blocks from Prometheus don't have any external labels stored in them. +Only blocks from Thanos use labels. +{{< /admonition >}} ## Considerations on Thanos specific features @@ -93,7 +97,9 @@ This may cause that incorrect results are returned for the query. Create an intermediate object storage bucket (such as Amazon S3 or GCS) within your cloud provider, where you can copy the historical blocks and work on them before uploading them to the Mimir bucket. - > **Tip:** Run the commands within a `screen` or `tmux` session, to avoid any interruptions because the steps might take time depending on the amount of data being processed. + {{< admonition type="tip" >}} + Run the commands within a `screen` or `tmux` session, to avoid any interruptions because the steps might take some time depending on the amount of data. + {{< /admonition >}} For Amazon S3, use the `aws` tool: @@ -161,7 +167,9 @@ This may cause that incorrect results are returned for the query. done ``` - > **Note:** Replace `prometheus_replica` with the unique label that would differentiate Prometheus replicas in your setup. + {{< admonition type="note" >}} + You must replace `prometheus_replica` with the unique label that would differentiate Prometheus replicas in your setup. + {{< /admonition >}} Clean up the duplicate blocks marked for deletion again: @@ -171,19 +179,24 @@ This may cause that incorrect results are returned for the query. --delete-delay=0 ``` - > **Tip:** If you want to visualize exactly what is happening in the blocks, with respect to the source of blocks, external labels, compaction levels, and more, you can use the following command to get the output as CSV and import it into a Google spreadsheet: - > - > ```bash - > thanos tools bucket inspect \ - > --objstore.config-file bucket-prod.yaml \ - > --output=csv > thanos-blocks.csv - > ``` + {{< admonition type="tip" >}} + If you want to visualize exactly what's happening in the blocks, with respect to the source of blocks, external labels, compaction levels, and more, you can use the following command to get the output as CSV and import it into a spreadsheet: + + ```bash + thanos tools bucket inspect \ + --objstore.config-file bucket-prod.yaml \ + --output=csv > thanos-blocks.csv + ``` + + {{< /admonition >}} 1. Relabel the blocks with external labels. Mimir doesn’t inject external labels from the `meta.json` file into query results. Therefore, you need to relabel the blocks with the required external labels in the `meta.json` file. - > **Note:** You can get the external labels in the `meta.json` file of each block from the CSV file that is imported (as mentioned in the preceding tip), and build the rewrite configuration accordingly. + {{< admonition type="tip" >}} + You can get the external labels in the `meta.json` file of each block from the CSV file that's imported, and build the rewrite configuration accordingly. + {{< /admonition >}} Create a rewrite configuration that is similar to this: @@ -261,28 +274,32 @@ This may cause that incorrect results are returned for the query. --delete-delay=0 ``` - > **Note**: If there are multiple prometheus clusters, then relabelling each of them in parallel may speed up the entire process. - > Get the list of blocks that for each cluster, and process it separately. - > - > ```bash - > thanos tools bucket inspect \ - > --objstore.config-file bucket.yaml \ - > --output=tsv \ - > | grep \ - > | awk '{print $1}' > prod-blocks.tsv - > ``` - - > ```bash - > for ID in `cat prod-blocks.tsv` - > do - > thanos tools bucket rewrite \ - > --objstore.config-file bucket.yaml \ - > --rewrite.to-relabel-config-file relabel-config.yaml \ - > --delete-blocks \ - > --no-dry-run \ - > --id $ID - > done - > ``` + {{< admonition type="note" >}} + If there are multiple Prometheus clusters, then relabelling each of them in parallel may speed up the entire process. + + Get the list of blocks that for each cluster, and process it separately. + + ```bash + thanos tools bucket inspect \ + --objstore.config-file bucket.yaml \ + --output=tsv \ + | grep \ + | awk '{print $1}' > prod-blocks.tsv + ``` + + ```bash + for ID in `cat prod-blocks.tsv` + do + thanos tools bucket rewrite \ + --objstore.config-file bucket.yaml \ + --rewrite.to-relabel-config-file relabel-config.yaml \ + --delete-blocks \ + --no-dry-run \ + --id $ID + done + ``` + + {{< /admonition >}} 1. Remove external labels from meta.json.