From 3b474f4acded59624519885f790369631fbe06a6 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Fri, 8 Oct 2021 13:59:24 -0400 Subject: [PATCH 01/26] Initial draft of Prometheus <-> OTLP datamodel specification. --- specification/metrics/datamodel.md | 115 +++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index cc70ec0923c..dd7b33cdb6b 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1013,6 +1013,121 @@ For comparison, see the simple logic used in [statsd sums](https://github.com/statsd/statsd/blob/master/stats.js#L281) where all points are added, and lost points are ignored. +## Prometheus Compatibility + +**Status**: [Experimental](../document-status.md) + +This section denotes how to convert from prometheus scraped metrics to the +OpenTelemtery metric data model and how to create prometheus metrics from +OpenTelemetry metric data. + +### Label Mapping + +Prometheus metric labels are split in opentelemetry across Resource attributes +and Metric data stream attributes. Some labels are used within metric +families to denote semantics which open-telemetry captures within the structure +of a data point. When mapping from prometheus to OpenTelemetry, any label +that is not explicitly called out as being handled specially will be included +in the set of attributes for a metric data stream. + +Here is a table of the sett of prometheus labels that are lifted into Resource +attributes when converting into OpenTelemetry. + +| Prometheus Label | OTLP Resource Attribute | Description | +| -------------------- | ----------------------- | ----------- | +| `job` | `service.name` | [Semantic convention](../resource/semantic_conventions/README.md#Service) | +| `host` | `host.name` | [Semantic convention](../resource/semantic_conventions/host.md) | +| `job` | `job` | ... | +| `instance` | `instance` | ... | +| `port` | `port` | ... | +| `__scheme__` | `scheme` | ... | + +Next, this set of attributes are "special" and used when converting from a +metric family to a specific OTLP metric data point: + +- `le` label is used to identify histogram bucket boundaries and counts. +- `quantile` label is used to identify quantile points in summary metrics. +- `__name__` is used to identify the metric name of the data point. +- `__metrics_path__` is ignored in OTLP. + +### Prometheus Metric points to OTLP + +Prometheus allows metrics to reported in "metric family" groups. While +OpenTelemetry can assume some shape/structure to metric family groups, any +metric belonging to a family that is not treated specially should be exported +as its own metric stream. For example, while histograms are expected to +live in a metric family with metrics `{name}_sum`, `{name}_count` and `{name}`, +any other metric also reported in the family should be exported as an +independent metric in OpenTelemetry. + +TODO - A bit about detecting/using start_time. + +Prometheus Counter becomes an OTLP Sum. + +Prometheus Gauge becomes an OTLP Gauge. + +Prometheus Unknown becomes an OTLP Gauge. + +Prometheus Histogram becomes an OTLP Histogram. + +Prometheus Summary becomes an OTLP Summary. + +Prometheus Gauge Histogram is dropped (TBD). + +Prometheus Stateset is dropped (TBD). + +Prometheus Info is dropped (TBD). + +### OTLP Metric points to Prometheus + +OpenTelemetry Gauge becomes a Prometheus Gauge. + +TODO: Example Gauge Conversions + +OpenTelemetry Sum follows this logic: + +- If the aggregation temporality is cumulative and the sum is monotonic, + then it becomes a Prometheus Sum. +- Otherwise the Sum becomes a Prometheus Gauge. + +TODO: Example Sum Conversions + +OpenTelemetry Histogram becomes a metric family with the following: + +- A single `{name}_count` metric denoting the count field of the histogram. + All attributes of the histogram point are converted to prometheus labels. +- `{name}_sum` metric denoting the sum field of the histogram, reported + only if the sum is positive and monotonic. All attributes of the histogram + point are converted to prometheus labels. +- A series of `{name}` metric points that contain all attributes of the + histogram point recorded as labels. Additionally, a label, denoted as `le` + is added denoting a bucket boundary, and having its value be the stringified + floating point value of bucket boundaries, starting form lowest to highest, + and all being non-negative. The value of each point is the sum of the count + of all histogram buckets up the the boundary reported in the `le` label. + These points will include a single exemplar that falls within `le` label and + no other `le` labelled point. + +_Note: OpenTelemetry DELTA histograms are not exported to prometheus._ + +TODO: Example Histogram conversion + +OpenTelemetry Summary becomes a metric family with the following: + +- A single `{name}_count` metric denoting the count field of the summary. + All attributes of the summary point are converted to prometheus labels. +- `{name}_sum` metric denoting the sum field of the summary, reported + only if the sum is positive and monotonic. All attributes of the summary + point are converted to prometheus labels. +- A series of `{name}` metric points that contain all attributes of the + summary point recorded as labels. Additionally, a label, denoted as + `quantile` is added denoting a reported qunatile point, and having its value + be the stringified floating point value of quantiles (between 0.0 and 1.0), + starting from lowest to highest, and all being non-negative. The value of + each point is the computed value of the quantile point. + +TODO: Example Summary conversion + ## Footnotes [1]: OTLP supports data point kinds that do not From 676fbdf94ba3dfa22ee799ae2f7c1f5261955e09 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Thu, 14 Oct 2021 13:08:32 -0400 Subject: [PATCH 02/26] Update specification/metrics/datamodel.md Co-authored-by: Joshua MacDonald --- specification/metrics/datamodel.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index dd7b33cdb6b..910816f225f 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1037,7 +1037,6 @@ attributes when converting into OpenTelemetry. | -------------------- | ----------------------- | ----------- | | `job` | `service.name` | [Semantic convention](../resource/semantic_conventions/README.md#Service) | | `host` | `host.name` | [Semantic convention](../resource/semantic_conventions/host.md) | -| `job` | `job` | ... | | `instance` | `instance` | ... | | `port` | `port` | ... | | `__scheme__` | `scheme` | ... | From a5262be7a023e44d05c25f2ee4949ce484b4a142 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Fri, 22 Oct 2021 16:29:11 -0400 Subject: [PATCH 03/26] Add blurb about labels and incompatibilities. --- specification/metrics/datamodel.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 910816f225f..bb81e38615d 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1023,7 +1023,7 @@ OpenTelemetry metric data. ### Label Mapping -Prometheus metric labels are split in opentelemetry across Resource attributes +Prometheus metric labels are split in OpenTelemetry across Resource attributes and Metric data stream attributes. Some labels are used within metric families to denote semantics which open-telemetry captures within the structure of a data point. When mapping from prometheus to OpenTelemetry, any label @@ -1049,6 +1049,11 @@ metric family to a specific OTLP metric data point: - `__name__` is used to identify the metric name of the data point. - `__metrics_path__` is ignored in OTLP. +Additionally, in Prometheus metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics +from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This +may cause ambiguity in scenarios where mulitple similar-named attributes share invalid characters at the same +location. This is considered an unsupported case, and is highly unlikely. + ### Prometheus Metric points to OTLP Prometheus allows metrics to reported in "metric family" groups. While From a066d974a4d1786576a5d411caedaa735391d916 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 14 Jan 2022 11:59:24 -0800 Subject: [PATCH 04/26] address minor review comments, and fix presubmit failures --- specification/metrics/datamodel.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index bb81e38615d..810439a2d42 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -45,6 +45,10 @@ * [Sums: Delta-to-Cumulative](#sums-delta-to-cumulative) + [Sums: detecting alignment issues](#sums-detecting-alignment-issues) + [Sums: Missing Timestamps](#sums-missing-timestamps) +- [Prometheus Compatibility](#prometheus-compatibility) + * [Label Mapping](#label-mapping) + * [Prometheus Metric points to OTLP](#prometheus-metric-points-to-otlp) + * [OTLP Metric points to Prometheus](#otlp-metric-points-to-prometheus) - [Footnotes](#footnotes) @@ -1018,25 +1022,25 @@ where all points are added, and lost points are ignored. **Status**: [Experimental](../document-status.md) This section denotes how to convert from prometheus scraped metrics to the -OpenTelemtery metric data model and how to create prometheus metrics from +OpenTelemetry metric data model and how to create Prometheus metrics from OpenTelemetry metric data. ### Label Mapping Prometheus metric labels are split in OpenTelemetry across Resource attributes and Metric data stream attributes. Some labels are used within metric -families to denote semantics which open-telemetry captures within the structure +families to denote semantics which OpenTelemetry captures within the structure of a data point. When mapping from prometheus to OpenTelemetry, any label that is not explicitly called out as being handled specially will be included in the set of attributes for a metric data stream. -Here is a table of the sett of prometheus labels that are lifted into Resource +Here is a table of the set of prometheus labels that are lifted into Resource attributes when converting into OpenTelemetry. | Prometheus Label | OTLP Resource Attribute | Description | | -------------------- | ----------------------- | ----------- | -| `job` | `service.name` | [Semantic convention](../resource/semantic_conventions/README.md#Service) | -| `host` | `host.name` | [Semantic convention](../resource/semantic_conventions/host.md) | +| `job` | `service.name` | [Semantic convention](../../semantic_conventions/resource/service.yaml) | +| `host` | `host.name` | [Semantic convention](../../semantic_conventions/trace/general.yaml) | | `instance` | `instance` | ... | | `port` | `port` | ... | | `__scheme__` | `scheme` | ... | @@ -1051,8 +1055,8 @@ metric family to a specific OTLP metric data point: Additionally, in Prometheus metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This -may cause ambiguity in scenarios where mulitple similar-named attributes share invalid characters at the same -location. This is considered an unsupported case, and is highly unlikely. +may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same +location. This is considered an unsupported case, and is highly unlikely. ### Prometheus Metric points to OTLP From f6e121281fb583fbf5e996bd8a122cea424557eb Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 14 Jan 2022 14:26:07 -0800 Subject: [PATCH 05/26] update prometheus SD labels based on current state, and update formatting --- specification/metrics/datamodel.md | 59 +++++++++++++----------------- 1 file changed, 26 insertions(+), 33 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 810439a2d42..7d4f7e7b351 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -46,9 +46,10 @@ + [Sums: detecting alignment issues](#sums-detecting-alignment-issues) + [Sums: Missing Timestamps](#sums-missing-timestamps) - [Prometheus Compatibility](#prometheus-compatibility) - * [Label Mapping](#label-mapping) + * [Prometheus Labels from Service Discovery](#prometheus-labels-from-service-discovery) * [Prometheus Metric points to OTLP](#prometheus-metric-points-to-otlp) * [OTLP Metric points to Prometheus](#otlp-metric-points-to-prometheus) + + [OpenTelemetry Attributes to Prometheus Labels](#opentelemetry-attributes-to-prometheus-labels) - [Footnotes](#footnotes) @@ -1025,49 +1026,24 @@ This section denotes how to convert from prometheus scraped metrics to the OpenTelemetry metric data model and how to create Prometheus metrics from OpenTelemetry metric data. -### Label Mapping +### Prometheus Labels from Service Discovery -Prometheus metric labels are split in OpenTelemetry across Resource attributes -and Metric data stream attributes. Some labels are used within metric -families to denote semantics which OpenTelemetry captures within the structure -of a data point. When mapping from prometheus to OpenTelemetry, any label -that is not explicitly called out as being handled specially will be included -in the set of attributes for a metric data stream. +When scraping a Prometheus endpoint, Prometheus service discovery discovers attributes from scraped targets prior to scraping them, and adds these as labels on incoming metrics. In OpenTelemetry, some of these labels are converted to Resource attributes, with the rest added as Metric data stream attributes. Here is a table of the set of prometheus labels that are lifted into Resource attributes when converting into OpenTelemetry. | Prometheus Label | OTLP Resource Attribute | Description | | -------------------- | ----------------------- | ----------- | -| `job` | `service.name` | [Semantic convention](../../semantic_conventions/resource/service.yaml) | -| `host` | `host.name` | [Semantic convention](../../semantic_conventions/trace/general.yaml) | -| `instance` | `instance` | ... | -| `port` | `port` | ... | +| `job` | `service.name` | ... | +| `job` | `job` | ... | +| `instance` | `instace` | ... | +| `instance` | `host.name` | instance is split into host:port | +| `instance` | `port` | instance is split into host:port | | `__scheme__` | `scheme` | ... | -Next, this set of attributes are "special" and used when converting from a -metric family to a specific OTLP metric data point: - -- `le` label is used to identify histogram bucket boundaries and counts. -- `quantile` label is used to identify quantile points in summary metrics. -- `__name__` is used to identify the metric name of the data point. -- `__metrics_path__` is ignored in OTLP. - -Additionally, in Prometheus metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics -from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This -may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same -location. This is considered an unsupported case, and is highly unlikely. - ### Prometheus Metric points to OTLP -Prometheus allows metrics to reported in "metric family" groups. While -OpenTelemetry can assume some shape/structure to metric family groups, any -metric belonging to a family that is not treated specially should be exported -as its own metric stream. For example, while histograms are expected to -live in a metric family with metrics `{name}_sum`, `{name}_count` and `{name}`, -any other metric also reported in the family should be exported as an -independent metric in OpenTelemetry. - TODO - A bit about detecting/using start_time. Prometheus Counter becomes an OTLP Sum. @@ -1078,8 +1054,18 @@ Prometheus Unknown becomes an OTLP Gauge. Prometheus Histogram becomes an OTLP Histogram. +Multiple Prometheus metrics are merged together into a single metric: + +* The `le` label on non-suffixed metrics is used to identify histogram bucket boundaries. Each Prometheus line produces one bucket on the resulting histogram. +* Lines with `_count` and `_sum` suffixes are used to determine the histogram's count and sum. + Prometheus Summary becomes an OTLP Summary. +Multiple Prometheus metrics are merged together into a single metric: + +* The `quantile` label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus metrics produces one quantile on the resulting summary. +* Lines with `_count` and `_sum` suffixes are used to determine the summary's count and sum. + Prometheus Gauge Histogram is dropped (TBD). Prometheus Stateset is dropped (TBD). @@ -1136,6 +1122,13 @@ OpenTelemetry Summary becomes a metric family with the following: TODO: Example Summary conversion +#### OpenTelemetry Attributes to Prometheus Labels + +In Prometheus metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics +from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This +may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same +location. This is considered an unsupported case, and is highly unlikely. + ## Footnotes [1]: OTLP supports data point kinds that do not From 0f84de21bd73a77fc530c7620b55ed9d770633df Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 19 Jan 2022 12:46:35 -0800 Subject: [PATCH 06/26] handle start time and delta sums --- specification/metrics/datamodel.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 7d4f7e7b351..ae1306f370b 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -774,7 +774,7 @@ misbehaving system. Receivers SHOULD presume a single writer was intended and eliminate overlap / deduplicate. Note: Identity is an important concept in most metrics systems. For example, -[Prometheus directly calls out uniqueness](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs): +[Prometheus directly calls out uniqueness](https://Prometheus.io/docs/Prometheus/latest/configuration/configuration/#metric_relabel_configs): > Take care with `labeldrop` and `labelkeep` to ensure that metrics > are still uniquely labeled once the labels are removed. @@ -1022,7 +1022,7 @@ where all points are added, and lost points are ignored. **Status**: [Experimental](../document-status.md) -This section denotes how to convert from prometheus scraped metrics to the +This section denotes how to convert from Prometheus scraped metrics to the OpenTelemetry metric data model and how to create Prometheus metrics from OpenTelemetry metric data. @@ -1030,7 +1030,7 @@ OpenTelemetry metric data. When scraping a Prometheus endpoint, Prometheus service discovery discovers attributes from scraped targets prior to scraping them, and adds these as labels on incoming metrics. In OpenTelemetry, some of these labels are converted to Resource attributes, with the rest added as Metric data stream attributes. -Here is a table of the set of prometheus labels that are lifted into Resource +Here is a table of the set of Prometheus labels that are lifted into Resource attributes when converting into OpenTelemetry. | Prometheus Label | OTLP Resource Attribute | Description | @@ -1044,7 +1044,7 @@ attributes when converting into OpenTelemetry. ### Prometheus Metric points to OTLP -TODO - A bit about detecting/using start_time. +Prometheus Counter metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, the `process_start_time_metric` may be used to provide the start time. If the start time is not provided by that mechanism, Prometheus conversion follows [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). Prometheus Counter becomes an OTLP Sum. @@ -1081,7 +1081,8 @@ TODO: Example Gauge Conversions OpenTelemetry Sum follows this logic: - If the aggregation temporality is cumulative and the sum is monotonic, - then it becomes a Prometheus Sum. + it becomes a Prometheus Sum. +- If the aggregation temporality is delta and the sum is monotonic, it may be converted to a cumulative temporality and become a Prometheus Sum, or it may be dropped. - Otherwise the Sum becomes a Prometheus Gauge. TODO: Example Sum Conversions @@ -1089,10 +1090,10 @@ TODO: Example Sum Conversions OpenTelemetry Histogram becomes a metric family with the following: - A single `{name}_count` metric denoting the count field of the histogram. - All attributes of the histogram point are converted to prometheus labels. + All attributes of the histogram point are converted to Prometheus labels. - `{name}_sum` metric denoting the sum field of the histogram, reported only if the sum is positive and monotonic. All attributes of the histogram - point are converted to prometheus labels. + point are converted to Prometheus labels. - A series of `{name}` metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted as `le` is added denoting a bucket boundary, and having its value be the stringified @@ -1102,17 +1103,17 @@ OpenTelemetry Histogram becomes a metric family with the following: These points will include a single exemplar that falls within `le` label and no other `le` labelled point. -_Note: OpenTelemetry DELTA histograms are not exported to prometheus._ +_Note: OpenTelemetry DELTA histograms are not exported to Prometheus._ TODO: Example Histogram conversion OpenTelemetry Summary becomes a metric family with the following: - A single `{name}_count` metric denoting the count field of the summary. - All attributes of the summary point are converted to prometheus labels. + All attributes of the summary point are converted to Prometheus labels. - `{name}_sum` metric denoting the sum field of the summary, reported only if the sum is positive and monotonic. All attributes of the summary - point are converted to prometheus labels. + point are converted to Prometheus labels. - A series of `{name}` metric points that contain all attributes of the summary point recorded as labels. Additionally, a label, denoted as `quantile` is added denoting a reported qunatile point, and having its value From a05a7c64546875fc04031a4ae92ecce11be2c91f Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 19 Jan 2022 14:46:30 -0800 Subject: [PATCH 07/26] add links and headers. Handle exemplars and otel resource attributes. --- specification/metrics/datamodel.md | 87 ++++++++++++++++++++++-------- 1 file changed, 65 insertions(+), 22 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index ae1306f370b..ede15415ac7 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -48,8 +48,23 @@ - [Prometheus Compatibility](#prometheus-compatibility) * [Prometheus Labels from Service Discovery](#prometheus-labels-from-service-discovery) * [Prometheus Metric points to OTLP](#prometheus-metric-points-to-otlp) + + [Counters](#counters) + + [Gauges](#gauges) + + [Unknown-typed](#unknown-typed) + + [Histograms](#histograms) + + [Summaries](#summaries) + + [Dropped Types](#dropped-types) + + [Start Time](#start-time) + + [Exemplars](#exemplars-1) * [OTLP Metric points to Prometheus](#otlp-metric-points-to-prometheus) - + [OpenTelemetry Attributes to Prometheus Labels](#opentelemetry-attributes-to-prometheus-labels) + + [Gauges](#gauges-1) + + [Sums](#sums-1) + + [Histograms](#histograms-1) + + [Summaries](#summaries-1) + + [Dropped Types](#dropped-types-1) + + [OpenTelemetry Metric Attributes](#opentelemetry-metric-attributes) + + [OpenTelemetry Exemplars](#opentelemetry-exemplars) + + [OpenTelemetry Resource Attributes](#opentelemetry-resource-attributes) - [Footnotes](#footnotes) @@ -1022,9 +1037,9 @@ where all points are added, and lost points are ignored. **Status**: [Experimental](../document-status.md) -This section denotes how to convert from Prometheus scraped metrics to the +This section denotes how to convert metrics scraped in the [Prometheus exposition](https://github.com/Showmax/prometheus-docs/blob/master/content/docs/instrumenting/exposition_formats.md#exposition-formats) or [OpenMetrics](https://openmetrics.io/) formats to the OpenTelemetry metric data model and how to create Prometheus metrics from -OpenTelemetry metric data. +OpenTelemetry metric data. Since OpenMetrics has a superset of Prometheus' types, "Prometheus" should be taken to mean "Prometheus or OpenMetrics". "OpenMetrics" refers to OpenMetrics-only concepts. ### Prometheus Labels from Service Discovery @@ -1044,50 +1059,68 @@ attributes when converting into OpenTelemetry. ### Prometheus Metric points to OTLP -Prometheus Counter metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, the `process_start_time_metric` may be used to provide the start time. If the start time is not provided by that mechanism, Prometheus conversion follows [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). +#### Counters -Prometheus Counter becomes an OTLP Sum. +[Prometheus Counter](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#counter) becomes an OTLP Sum. -Prometheus Gauge becomes an OTLP Gauge. +#### Gauges -Prometheus Unknown becomes an OTLP Gauge. +[Prometheus Gauge](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#gauge) becomes an OTLP Gauge. -Prometheus Histogram becomes an OTLP Histogram. +#### Unknown-typed + +[Prometheus Unknown](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#unknown) becomes an OTLP Gauge. + +#### Histograms + +[Prometheus Histogram](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#histogram) becomes an OTLP Histogram. Multiple Prometheus metrics are merged together into a single metric: * The `le` label on non-suffixed metrics is used to identify histogram bucket boundaries. Each Prometheus line produces one bucket on the resulting histogram. * Lines with `_count` and `_sum` suffixes are used to determine the histogram's count and sum. -Prometheus Summary becomes an OTLP Summary. +#### Summaries + +[Prometheus Summary](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#summary) becomes an OTLP Summary. Multiple Prometheus metrics are merged together into a single metric: * The `quantile` label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus metrics produces one quantile on the resulting summary. * Lines with `_count` and `_sum` suffixes are used to determine the summary's count and sum. -Prometheus Gauge Histogram is dropped (TBD). +#### Dropped Types + +* [OpenMetrics GaugeHistogram](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#gaugehistogram) +* [OpenMetrics StateSet](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#stateset) +* [OpenMetrics Info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#info) + +#### Start Time + +Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, the `process_start_time_metric` may be used to provide the start time. If the start time is not provided by that mechanism, Prometheus conversion follows [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). -Prometheus Stateset is dropped (TBD). +#### Exemplars -Prometheus Info is dropped (TBD). +[Prometheus Exemplars](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars) may be attached to Prometheus Histogram bucket metrics, which should be converted to exemplars on OpenTelemetry histograms. The Trace ID and Span ID should be retrieved from the `trace_id` and `span_id` label keys, respectively. ### OTLP Metric points to Prometheus +#### Gauges + OpenTelemetry Gauge becomes a Prometheus Gauge. -TODO: Example Gauge Conversions +#### Sums OpenTelemetry Sum follows this logic: - If the aggregation temporality is cumulative and the sum is monotonic, it becomes a Prometheus Sum. -- If the aggregation temporality is delta and the sum is monotonic, it may be converted to a cumulative temporality and become a Prometheus Sum, or it may be dropped. +- If the aggregation temporality is delta and the sum is monotonic, it should be converted to a cumulative temporality and become a Prometheus Sum, or it may be dropped. - Otherwise the Sum becomes a Prometheus Gauge. -TODO: Example Sum Conversions +#### Histograms -OpenTelemetry Histogram becomes a metric family with the following: +OpenTelemetry Histogram with a cumulative aggregation temporality becomes a metric family with the following: - A single `{name}_count` metric denoting the count field of the histogram. All attributes of the histogram point are converted to Prometheus labels. @@ -1103,9 +1136,9 @@ OpenTelemetry Histogram becomes a metric family with the following: These points will include a single exemplar that falls within `le` label and no other `le` labelled point. -_Note: OpenTelemetry DELTA histograms are not exported to Prometheus._ +OpenTelemetry Histograms with Delta aggregation temporality should be aggregated into a Cumulative aggregation temporality, or may be dropped. -TODO: Example Histogram conversion +#### Summaries OpenTelemetry Summary becomes a metric family with the following: @@ -1121,14 +1154,24 @@ OpenTelemetry Summary becomes a metric family with the following: starting from lowest to highest, and all being non-negative. The value of each point is the computed value of the quantile point. -TODO: Example Summary conversion +#### Dropped Types + +* ExponentialHistogram -#### OpenTelemetry Attributes to Prometheus Labels +#### OpenTelemetry Metric Attributes -In Prometheus metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics +OpenTelemetry Metric Attributes are converted to [Prometheus labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same -location. This is considered an unsupported case, and is highly unlikely. +location. In such unlikely cases, either value may be used. + +#### OpenTelemetry Exemplars + +Exemplars on OpenTelemetry Histograms should be converted to Prometheus exemplars. Exemplars on other OpenTelemetry metric types are dropped. For Prometheus push exporters, multiple exemplars are permitted on each bucket, so all exemplars are converted. For Prometheus pull endpoints, only a single exemplar can be added to each bucket, so the largest exemplar from each bucket is chosen. If no exemplars exist on a bucket, the highest exemplar from a lower bucket should be used, even if it is a duplicate of another bucket. Prometheus Exemplars should use the `trace_id` and `span_id` keys for the trace and span IDs, respectively. + +#### OpenTelemetry Resource Attributes + +In the collector, but not in SDKs, only the `job` and `instance` resource attributes are converted to Prometheus metric labels. In all other cases, resource attributes are dropped, and are not attached to Prometheus metrics. ## Footnotes From c0ffb1fdc41e222741ba844baeef9b9efd55b47d Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 19 Jan 2022 19:48:21 -0500 Subject: [PATCH 08/26] Apply suggestions from code review Co-authored-by: Anthony Mirabella --- specification/metrics/datamodel.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index ede15415ac7..c6bce138dba 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -789,7 +789,7 @@ misbehaving system. Receivers SHOULD presume a single writer was intended and eliminate overlap / deduplicate. Note: Identity is an important concept in most metrics systems. For example, -[Prometheus directly calls out uniqueness](https://Prometheus.io/docs/Prometheus/latest/configuration/configuration/#metric_relabel_configs): +[Prometheus directly calls out uniqueness](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs): > Take care with `labeldrop` and `labelkeep` to ensure that metrics > are still uniquely labeled once the labels are removed. @@ -1052,7 +1052,7 @@ attributes when converting into OpenTelemetry. | -------------------- | ----------------------- | ----------- | | `job` | `service.name` | ... | | `job` | `job` | ... | -| `instance` | `instace` | ... | +| `instance` | `instance` | ... | | `instance` | `host.name` | instance is split into host:port | | `instance` | `port` | instance is split into host:port | | `__scheme__` | `scheme` | ... | @@ -1086,7 +1086,7 @@ Multiple Prometheus metrics are merged together into a single metric: Multiple Prometheus metrics are merged together into a single metric: -* The `quantile` label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus metrics produces one quantile on the resulting summary. +* The `quantile` label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus line produces one quantile on the resulting summary. * Lines with `_count` and `_sum` suffixes are used to determine the summary's count and sum. #### Dropped Types From 163f9056ac34be5203d4d258eab9892793a79962 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 19 Jan 2022 17:02:24 -0800 Subject: [PATCH 09/26] fix link --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index c6bce138dba..8dbfda46086 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1037,7 +1037,7 @@ where all points are added, and lost points are ignored. **Status**: [Experimental](../document-status.md) -This section denotes how to convert metrics scraped in the [Prometheus exposition](https://github.com/Showmax/prometheus-docs/blob/master/content/docs/instrumenting/exposition_formats.md#exposition-formats) or [OpenMetrics](https://openmetrics.io/) formats to the +This section denotes how to convert metrics scraped in the [Prometheus exposition](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exposition-formats) or [OpenMetrics](https://openmetrics.io/) formats to the OpenTelemetry metric data model and how to create Prometheus metrics from OpenTelemetry metric data. Since OpenMetrics has a superset of Prometheus' types, "Prometheus" should be taken to mean "Prometheus or OpenMetrics". "OpenMetrics" refers to OpenMetrics-only concepts. From f82330634b7f1f8d1c6eea366e24e7eb3936d4d5 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 21 Jan 2022 12:17:07 -0800 Subject: [PATCH 10/26] change prom service discovery section to resource attribute sections --- specification/metrics/datamodel.md | 55 +++++++++++++++--------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 8dbfda46086..5b2efd10ef4 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -46,7 +46,6 @@ + [Sums: detecting alignment issues](#sums-detecting-alignment-issues) + [Sums: Missing Timestamps](#sums-missing-timestamps) - [Prometheus Compatibility](#prometheus-compatibility) - * [Prometheus Labels from Service Discovery](#prometheus-labels-from-service-discovery) * [Prometheus Metric points to OTLP](#prometheus-metric-points-to-otlp) + [Counters](#counters) + [Gauges](#gauges) @@ -56,15 +55,16 @@ + [Dropped Types](#dropped-types) + [Start Time](#start-time) + [Exemplars](#exemplars-1) + + [Resource Attributes](#resource-attributes) * [OTLP Metric points to Prometheus](#otlp-metric-points-to-prometheus) + [Gauges](#gauges-1) + [Sums](#sums-1) + [Histograms](#histograms-1) + [Summaries](#summaries-1) + [Dropped Types](#dropped-types-1) - + [OpenTelemetry Metric Attributes](#opentelemetry-metric-attributes) - + [OpenTelemetry Exemplars](#opentelemetry-exemplars) - + [OpenTelemetry Resource Attributes](#opentelemetry-resource-attributes) + + [Metric Attributes](#metric-attributes) + + [Exemplars](#exemplars-2) + + [Resource Attributes](#resource-attributes-1) - [Footnotes](#footnotes) @@ -1037,25 +1037,9 @@ where all points are added, and lost points are ignored. **Status**: [Experimental](../document-status.md) -This section denotes how to convert metrics scraped in the [Prometheus exposition](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exposition-formats) or [OpenMetrics](https://openmetrics.io/) formats to the +This section denotes how to convert metrics scraped in the [Prometheus exposition](https://github.com/Prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#exposition-formats) or [OpenMetrics](https://openmetrics.io/) formats to the OpenTelemetry metric data model and how to create Prometheus metrics from -OpenTelemetry metric data. Since OpenMetrics has a superset of Prometheus' types, "Prometheus" should be taken to mean "Prometheus or OpenMetrics". "OpenMetrics" refers to OpenMetrics-only concepts. - -### Prometheus Labels from Service Discovery - -When scraping a Prometheus endpoint, Prometheus service discovery discovers attributes from scraped targets prior to scraping them, and adds these as labels on incoming metrics. In OpenTelemetry, some of these labels are converted to Resource attributes, with the rest added as Metric data stream attributes. - -Here is a table of the set of Prometheus labels that are lifted into Resource -attributes when converting into OpenTelemetry. - -| Prometheus Label | OTLP Resource Attribute | Description | -| -------------------- | ----------------------- | ----------- | -| `job` | `service.name` | ... | -| `job` | `job` | ... | -| `instance` | `instance` | ... | -| `instance` | `host.name` | instance is split into host:port | -| `instance` | `port` | instance is split into host:port | -| `__scheme__` | `scheme` | ... | +OpenTelemetry metric data. Since OpenMetrics has a superset of Prometheus' types, "Prometheus" is taken to mean "Prometheus or OpenMetrics". "OpenMetrics" refers to OpenMetrics-only concepts. ### Prometheus Metric points to OTLP @@ -1103,6 +1087,21 @@ Prometheus Cumulative metrics do not include the start time of the metric. When [Prometheus Exemplars](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars) may be attached to Prometheus Histogram bucket metrics, which should be converted to exemplars on OpenTelemetry histograms. The Trace ID and Span ID should be retrieved from the `trace_id` and `span_id` label keys, respectively. +#### Resource Attributes + +When scraping a Prometheus endpoint, resource attributes must be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints. In particular, `job` and `instance`, [as defined by Prometheus](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances), are needed to ensure Prometheus exporters can disambiguate metrics as [described below](#resource-attributes-1). + +The following resource attributes must be added to scraped metrics, and may not be added as metric attributes: + +| OTLP Resource Attribute | Description | +| ----------------------- | ----------- | +| `service.name` | The configured name of the service that the target belongs to | +| `job` | Identical to `service.name` | +| `instance` | The : of the target's URL that was scraped. | +| `host.name` | `instance` is split into : | +| `port` | `instance` is split into : | +| `scheme` | `http` or `https` | + ### OTLP Metric points to Prometheus #### Gauges @@ -1158,20 +1157,22 @@ OpenTelemetry Summary becomes a metric family with the following: * ExponentialHistogram -#### OpenTelemetry Metric Attributes +#### Metric Attributes -OpenTelemetry Metric Attributes are converted to [Prometheus labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics +OpenTelemetry Metric Attributes are converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same location. In such unlikely cases, either value may be used. -#### OpenTelemetry Exemplars +#### Exemplars Exemplars on OpenTelemetry Histograms should be converted to Prometheus exemplars. Exemplars on other OpenTelemetry metric types are dropped. For Prometheus push exporters, multiple exemplars are permitted on each bucket, so all exemplars are converted. For Prometheus pull endpoints, only a single exemplar can be added to each bucket, so the largest exemplar from each bucket is chosen. If no exemplars exist on a bucket, the highest exemplar from a lower bucket should be used, even if it is a duplicate of another bucket. Prometheus Exemplars should use the `trace_id` and `span_id` keys for the trace and span IDs, respectively. -#### OpenTelemetry Resource Attributes +#### Resource Attributes + +In SDK Prometheus (pull) exporters, all resource attributes are dropped, and are not attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping. -In the collector, but not in SDKs, only the `job` and `instance` resource attributes are converted to Prometheus metric labels. In all other cases, resource attributes are dropped, and are not attached to Prometheus metrics. +In the Collector's Prometheus pull and push (remote-write) exporters, metrics from multiple targets may be sent together, and must be disambiguated from one another. However, the Prometheus exposition format and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto) formats do not include a notion of resource, and expect metric labels to distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances) labels distinguish targets and are expected to be present on metrics exposed on a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/) Prometheus endpoint) or pushed via Prometheus remote-write. In the collector, only the `job` and `instance` resource attributes are converted to Prometheus metric labels. ## Footnotes From 77a0a1085fb56c8c51dc03e744d29912fa319f14 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 27 Jan 2022 12:40:30 -0800 Subject: [PATCH 11/26] disallow process_start_time_seconds --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 5b2efd10ef4..47c59391915 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1081,7 +1081,7 @@ Multiple Prometheus metrics are merged together into a single metric: #### Start Time -Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, the `process_start_time_metric` may be used to provide the start time. If the start time is not provided by that mechanism, Prometheus conversion follows [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). +Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion must follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). #### Exemplars From d6c52453a62dbd872c98a858022d61446e624ede Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 28 Jan 2022 06:37:27 -0800 Subject: [PATCH 12/26] add links for otel data points --- specification/metrics/datamodel.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 47c59391915..e484dfbb5e8 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -61,7 +61,7 @@ + [Sums](#sums-1) + [Histograms](#histograms-1) + [Summaries](#summaries-1) - + [Dropped Types](#dropped-types-1) + + [Dropped Data Points](#dropped-data-points) + [Metric Attributes](#metric-attributes) + [Exemplars](#exemplars-2) + [Resource Attributes](#resource-attributes-1) @@ -1106,11 +1106,11 @@ The following resource attributes must be added to scraped metrics, and may not #### Gauges -OpenTelemetry Gauge becomes a Prometheus Gauge. +[OpenTelemetry Gauges](#gauge) becomes a Prometheus Gauge. #### Sums -OpenTelemetry Sum follows this logic: +[OpenTelemetry Sums](#sums) follows this logic: - If the aggregation temporality is cumulative and the sum is monotonic, it becomes a Prometheus Sum. @@ -1119,7 +1119,7 @@ OpenTelemetry Sum follows this logic: #### Histograms -OpenTelemetry Histogram with a cumulative aggregation temporality becomes a metric family with the following: +[OpenTelemetry Histograms](#histogram) with a cumulative aggregation temporality becomes a metric family with the following: - A single `{name}_count` metric denoting the count field of the histogram. All attributes of the histogram point are converted to Prometheus labels. @@ -1139,7 +1139,7 @@ OpenTelemetry Histograms with Delta aggregation temporality should be aggregated #### Summaries -OpenTelemetry Summary becomes a metric family with the following: +[OpenTelemetry Summary](#summary-legacy) becomes a metric family with the following: - A single `{name}_count` metric denoting the count field of the summary. All attributes of the summary point are converted to Prometheus labels. @@ -1153,9 +1153,9 @@ OpenTelemetry Summary becomes a metric family with the following: starting from lowest to highest, and all being non-negative. The value of each point is the computed value of the quantile point. -#### Dropped Types +#### Dropped Data Points -* ExponentialHistogram +* [ExponentialHistogram](#exponentialhistogram) #### Metric Attributes @@ -1166,7 +1166,7 @@ location. In such unlikely cases, either value may be used. #### Exemplars -Exemplars on OpenTelemetry Histograms should be converted to Prometheus exemplars. Exemplars on other OpenTelemetry metric types are dropped. For Prometheus push exporters, multiple exemplars are permitted on each bucket, so all exemplars are converted. For Prometheus pull endpoints, only a single exemplar can be added to each bucket, so the largest exemplar from each bucket is chosen. If no exemplars exist on a bucket, the highest exemplar from a lower bucket should be used, even if it is a duplicate of another bucket. Prometheus Exemplars should use the `trace_id` and `span_id` keys for the trace and span IDs, respectively. +[Exemplars](#exemplars) on OpenTelemetry Histograms should be converted to Prometheus exemplars. Exemplars on other OpenTelemetry data points are dropped. For Prometheus push exporters, multiple exemplars are permitted on each bucket, so all exemplars are converted. For Prometheus pull endpoints, only a single exemplar can be added to each bucket, so the largest exemplar from each bucket is chosen. If no exemplars exist on a bucket, the highest exemplar from a lower bucket should be used, even if it is a duplicate of another bucket. Prometheus Exemplars should use the `trace_id` and `span_id` keys for the trace and span IDs, respectively. #### Resource Attributes From 958d624d80cddf55c96d19864b3a01ae5deff6b0 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 3 Feb 2022 12:00:31 -0500 Subject: [PATCH 13/26] Apply suggestions from code review Co-authored-by: Tyler Yahn --- specification/metrics/datamodel.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index e484dfbb5e8..f074747adb0 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1091,7 +1091,7 @@ Prometheus Cumulative metrics do not include the start time of the metric. When When scraping a Prometheus endpoint, resource attributes must be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints. In particular, `job` and `instance`, [as defined by Prometheus](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances), are needed to ensure Prometheus exporters can disambiguate metrics as [described below](#resource-attributes-1). -The following resource attributes must be added to scraped metrics, and may not be added as metric attributes: +The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as attributes to the metric instruments themselves: | OTLP Resource Attribute | Description | | ----------------------- | ----------- | @@ -1129,7 +1129,7 @@ The following resource attributes must be added to scraped metrics, and may not - A series of `{name}` metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted as `le` is added denoting a bucket boundary, and having its value be the stringified - floating point value of bucket boundaries, starting form lowest to highest, + floating point value of bucket boundaries, starting from lowest to highest, and all being non-negative. The value of each point is the sum of the count of all histogram buckets up the the boundary reported in the `le` label. These points will include a single exemplar that falls within `le` label and From e49ef471f61c288ae5d0f170aea13408500732f5 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 3 Feb 2022 11:06:00 -0800 Subject: [PATCH 14/26] use normative language --- specification/metrics/datamodel.md | 56 ++++++++++++++++-------------- 1 file changed, 30 insertions(+), 26 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index f074747adb0..222cc8140c2 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1045,81 +1045,83 @@ OpenTelemetry metric data. Since OpenMetrics has a superset of Prometheus' types #### Counters -[Prometheus Counter](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#counter) becomes an OTLP Sum. +A [Prometheus Counter](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#counter) MUST be converted to an OTLP Sum. #### Gauges -[Prometheus Gauge](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#gauge) becomes an OTLP Gauge. +A [Prometheus Gauge](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#gauge) MUST be converted to an OTLP Gauge. #### Unknown-typed -[Prometheus Unknown](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#unknown) becomes an OTLP Gauge. +A [Prometheus Unknown](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#unknown) MUST be converted to an OTLP Gauge. #### Histograms -[Prometheus Histogram](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#histogram) becomes an OTLP Histogram. +A [Prometheus Histogram](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#histogram) MUST be converted to an OTLP Histogram. -Multiple Prometheus metrics are merged together into a single metric: +Multiple Prometheus histogram metrics MUST be merged together into a single OTLP Histogram: * The `le` label on non-suffixed metrics is used to identify histogram bucket boundaries. Each Prometheus line produces one bucket on the resulting histogram. * Lines with `_count` and `_sum` suffixes are used to determine the histogram's count and sum. #### Summaries -[Prometheus Summary](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#summary) becomes an OTLP Summary. +[Prometheus Summary](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#summary) MUST be converted to an OTLP Summary. -Multiple Prometheus metrics are merged together into a single metric: +Multiple Prometheus metrics are merged together into a single OTLP Summary: * The `quantile` label on non-suffixed metrics is used to identify quantile points in summary metrics. Each Prometheus line produces one quantile on the resulting summary. * Lines with `_count` and `_sum` suffixes are used to determine the summary's count and sum. #### Dropped Types +The following Prometheus types MUST be dropped: + * [OpenMetrics GaugeHistogram](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#gaugehistogram) * [OpenMetrics StateSet](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#stateset) * [OpenMetrics Info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#info) #### Start Time -Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion must follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). +Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion MUST follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). #### Exemplars -[Prometheus Exemplars](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars) may be attached to Prometheus Histogram bucket metrics, which should be converted to exemplars on OpenTelemetry histograms. The Trace ID and Span ID should be retrieved from the `trace_id` and `span_id` label keys, respectively. +[Prometheus Exemplars](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars) can be attached to Prometheus Histogram bucket metrics, which SHOULD be converted to exemplars on OpenTelemetry histograms. The Trace ID and Span ID SHOULD be retrieved from the `trace_id` and `span_id` label keys, respectively. #### Resource Attributes -When scraping a Prometheus endpoint, resource attributes must be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints. In particular, `job` and `instance`, [as defined by Prometheus](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances), are needed to ensure Prometheus exporters can disambiguate metrics as [described below](#resource-attributes-1). +When scraping a Prometheus endpoint, resource attributes MUST be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints. In particular, `job` and `instance`, [as defined by Prometheus](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances), are needed to ensure Prometheus exporters can disambiguate metrics as [described below](#resource-attributes-1). -The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as attributes to the metric instruments themselves: +The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes: | OTLP Resource Attribute | Description | | ----------------------- | ----------- | | `service.name` | The configured name of the service that the target belongs to | | `job` | Identical to `service.name` | | `instance` | The : of the target's URL that was scraped. | -| `host.name` | `instance` is split into : | -| `port` | `instance` is split into : | +| `host.name` | `instance` The portion of `instance` | +| `port` | `instance` The portion of `instance` | | `scheme` | `http` or `https` | ### OTLP Metric points to Prometheus #### Gauges -[OpenTelemetry Gauges](#gauge) becomes a Prometheus Gauge. +An [OpenTelemetry Gauge](#gauge) MUST be converted to a Prometheus Gauge. #### Sums [OpenTelemetry Sums](#sums) follows this logic: - If the aggregation temporality is cumulative and the sum is monotonic, - it becomes a Prometheus Sum. -- If the aggregation temporality is delta and the sum is monotonic, it should be converted to a cumulative temporality and become a Prometheus Sum, or it may be dropped. -- Otherwise the Sum becomes a Prometheus Gauge. + it MUST be converted to a Prometheus Sum. +- If the aggregation temporality is delta and the sum is monotonic, it SHOULD be converted to a cumulative temporality and become a Prometheus Sum, or it MAY be dropped. +- Otherwise the Sum MUST be converted to a Prometheus Gauge. #### Histograms -[OpenTelemetry Histograms](#histogram) with a cumulative aggregation temporality becomes a metric family with the following: +An [OpenTelemetry Histogram](#histogram) with a cumulative aggregation temporality MUST be converted to a Prometheus metric family with the following metrics: - A single `{name}_count` metric denoting the count field of the histogram. All attributes of the histogram point are converted to Prometheus labels. @@ -1135,11 +1137,11 @@ The following attributes MUST be associated with scraped metrics as resource att These points will include a single exemplar that falls within `le` label and no other `le` labelled point. -OpenTelemetry Histograms with Delta aggregation temporality should be aggregated into a Cumulative aggregation temporality, or may be dropped. +OpenTelemetry Histograms with Delta aggregation temporality SHOULD be aggregated into a Cumulative aggregation temporality and follow the logic above, or MAY be dropped. #### Summaries -[OpenTelemetry Summary](#summary-legacy) becomes a metric family with the following: +An [OpenTelemetry Summary](#summary-legacy) MUST be converted to a Prometheus metric family with the following metrics: - A single `{name}_count` metric denoting the count field of the summary. All attributes of the summary point are converted to Prometheus labels. @@ -1155,24 +1157,26 @@ OpenTelemetry Histograms with Delta aggregation temporality should be aggregated #### Dropped Data Points +The following OTLP data points must be dropped: + * [ExponentialHistogram](#exponentialhistogram) #### Metric Attributes -OpenTelemetry Metric Attributes are converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics -from OpenTelemetry with unsupported Attribute names should replace invalid characters with the `_` character. This +OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics +from OpenTelemetry with unsupported Attribute names SHOULD replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same -location. In such unlikely cases, either value may be used. +location. In such unlikely cases, either value MAY be used. #### Exemplars -[Exemplars](#exemplars) on OpenTelemetry Histograms should be converted to Prometheus exemplars. Exemplars on other OpenTelemetry data points are dropped. For Prometheus push exporters, multiple exemplars are permitted on each bucket, so all exemplars are converted. For Prometheus pull endpoints, only a single exemplar can be added to each bucket, so the largest exemplar from each bucket is chosen. If no exemplars exist on a bucket, the highest exemplar from a lower bucket should be used, even if it is a duplicate of another bucket. Prometheus Exemplars should use the `trace_id` and `span_id` keys for the trace and span IDs, respectively. +[Exemplars](#exemplars) on OpenTelemetry Histograms SHOULD be converted to Prometheus exemplars. Exemplars on other OpenTelemetry data points MUST be dropped. For Prometheus push exporters, multiple exemplars are able to be added to each bucket, so all exemplars SHOULD be converted. For Prometheus pull endpoints, only a single exemplar is able to be added to each bucket, so the largest exemplar from each bucket MUST be used, if attaching exemplars. If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket's exemplar. Prometheus Exemplars MUST use the `trace_id` and `span_id` keys for the trace and span IDs, respectively. #### Resource Attributes -In SDK Prometheus (pull) exporters, all resource attributes are dropped, and are not attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping. +In SDK Prometheus (pull) exporters, all resource attributes MUST be dropped, and MUST NOT be not attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping. -In the Collector's Prometheus pull and push (remote-write) exporters, metrics from multiple targets may be sent together, and must be disambiguated from one another. However, the Prometheus exposition format and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto) formats do not include a notion of resource, and expect metric labels to distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances) labels distinguish targets and are expected to be present on metrics exposed on a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/) Prometheus endpoint) or pushed via Prometheus remote-write. In the collector, only the `job` and `instance` resource attributes are converted to Prometheus metric labels. +In the Collector's Prometheus pull and push (remote-write) exporters, it is possible for metrics from multiple targets to be sent together, so targets must be disambiguated from one another. However, the Prometheus exposition format and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto) formats do not include a notion of resource, and expect metric labels to distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances) labels distinguish targets and are expected to be present on metrics exposed on a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/) Prometheus endpoint) or pushed via Prometheus remote-write. In the collector Prometheus exporters, the `job` and `instance` resource attributes MUST be converted to Prometheus metric labels, and other resource attributes SHOULD NOT be converted to metric labels. ## Footnotes From fc488e09d4385e593f474dac1de4cd609fd69b2f Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 3 Feb 2022 11:40:03 -0800 Subject: [PATCH 15/26] handle negative prometheus histogram buckets --- specification/metrics/datamodel.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 222cc8140c2..730dc8f2db1 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1063,6 +1063,8 @@ Multiple Prometheus histogram metrics MUST be merged together into a single OTLP * The `le` label on non-suffixed metrics is used to identify histogram bucket boundaries. Each Prometheus line produces one bucket on the resulting histogram. * Lines with `_count` and `_sum` suffixes are used to determine the histogram's count and sum. +* If `_count` is not present, the metric MUST be dropped. +* If `_sum` is not present, it MUST be computed from the buckets. #### Summaries @@ -1123,19 +1125,9 @@ An [OpenTelemetry Gauge](#gauge) MUST be converted to a Prometheus Gauge. An [OpenTelemetry Histogram](#histogram) with a cumulative aggregation temporality MUST be converted to a Prometheus metric family with the following metrics: -- A single `{name}_count` metric denoting the count field of the histogram. - All attributes of the histogram point are converted to Prometheus labels. -- `{name}_sum` metric denoting the sum field of the histogram, reported - only if the sum is positive and monotonic. All attributes of the histogram - point are converted to Prometheus labels. -- A series of `{name}` metric points that contain all attributes of the - histogram point recorded as labels. Additionally, a label, denoted as `le` - is added denoting a bucket boundary, and having its value be the stringified - floating point value of bucket boundaries, starting from lowest to highest, - and all being non-negative. The value of each point is the sum of the count - of all histogram buckets up the the boundary reported in the `le` label. - These points will include a single exemplar that falls within `le` label and - no other `le` labelled point. +- A single `{name}_count` metric denoting the count field of the histogram. All attributes of the histogram point are converted to Prometheus labels. +- `{name}_sum` metric denoting the sum field of the histogram, reported only if the sum is positive and monotonic. The sum is positive and monotonic when all buckets are positive. All attributes of the histogram point are converted to Prometheus labels. +- A series of `{name}` metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted as `le` is added denoting the bucket boundary. The label's value is the stringified floating point value of bucket boundaries, ordered from lowest to highest. The value of each point is the sum of the count of all histogram buckets up the the boundary reported in the `le` label. These points will include a single exemplar that falls within `le` label and no other `le` labelled point. The final bucket metric MUST have an `+Inf` threshold. OpenTelemetry Histograms with Delta aggregation temporality SHOULD be aggregated into a Cumulative aggregation temporality and follow the logic above, or MAY be dropped. From 6c02dafad56f264ecf3e72c78486c500f4269cef Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 3 Feb 2022 11:54:51 -0800 Subject: [PATCH 16/26] typo --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 730dc8f2db1..1a3b4c6e8d8 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1142,7 +1142,7 @@ An [OpenTelemetry Summary](#summary-legacy) MUST be converted to a Prometheus me point are converted to Prometheus labels. - A series of `{name}` metric points that contain all attributes of the summary point recorded as labels. Additionally, a label, denoted as - `quantile` is added denoting a reported qunatile point, and having its value + `quantile` is added denoting a reported quantile point, and having its value be the stringified floating point value of quantiles (between 0.0 and 1.0), starting from lowest to highest, and all being non-negative. The value of each point is the computed value of the quantile point. From 6fa9153df3a7738de31ebc7a5a37cc81f5699c14 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 3 Feb 2022 12:01:16 -0800 Subject: [PATCH 17/26] specify what to do with non-string attributes --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 1a3b4c6e8d8..e21e592a3fd 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1155,7 +1155,7 @@ The following OTLP data points must be dropped: #### Metric Attributes -OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics +OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). String Attribute values are converted directly to Metric Attributes, and non-string Attribute values SHOULD be replaced with an empty string. In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names SHOULD replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same location. In such unlikely cases, either value MAY be used. From 2e319206a32d5c064f76f21ef9e3dcb70d971714 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 8 Feb 2022 08:26:47 -0800 Subject: [PATCH 18/26] clarify sum behavior --- specification/metrics/datamodel.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index e21e592a3fd..e97b278cf80 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1116,10 +1116,10 @@ An [OpenTelemetry Gauge](#gauge) MUST be converted to a Prometheus Gauge. [OpenTelemetry Sums](#sums) follows this logic: -- If the aggregation temporality is cumulative and the sum is monotonic, - it MUST be converted to a Prometheus Sum. -- If the aggregation temporality is delta and the sum is monotonic, it SHOULD be converted to a cumulative temporality and become a Prometheus Sum, or it MAY be dropped. -- Otherwise the Sum MUST be converted to a Prometheus Gauge. +- If the aggregation temporality is cumulative and the sum is monotonic, it MUST be converted to a Prometheus Sum. +- If the aggregation temporality is cumulative and the sum is non-monotonic, it MUST be converted to a Prometheus Gauge. +- If the aggregation temporality is delta and the sum is monotonic, it SHOULD be converted to a cumulative temporality and become a Prometheus Sum +- Otherwise, it MUST be dropped. #### Histograms From 98b28b8ba261b2dff25bf177f3abee866b3e4a02 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 8 Feb 2022 08:39:03 -0800 Subject: [PATCH 19/26] handle label collisions when converting --- specification/metrics/datamodel.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index e97b278cf80..8d619f6f11b 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1155,10 +1155,7 @@ The following OTLP data points must be dropped: #### Metric Attributes -OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). String Attribute values are converted directly to Metric Attributes, and non-string Attribute values SHOULD be replaced with an empty string. In Prometheus, metric labels must match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics -from OpenTelemetry with unsupported Attribute names SHOULD replace invalid characters with the `_` character. This -may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same -location. In such unlikely cases, either value MAY be used. +OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). String Attribute values are converted directly to Metric Attributes, and non-string Attribute values SHOULD be replaced with an empty string. Prometheus metric label keys are required to match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names MUST replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same location. In such unlikely cases, if multiple key-value pairs are converted to have the same Prometheus key, the values MUST be concatenated together, separated by `;`, and ordered by the lexicographical order of the original keys. #### Exemplars From 0010b0d577160bc5934ff5c6722b06c9b655d9d8 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 9 Feb 2022 06:10:29 -0800 Subject: [PATCH 20/26] clarify that if deltas are not aggregated, they must be dropped --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 8d619f6f11b..bc49538901f 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1129,7 +1129,7 @@ An [OpenTelemetry Histogram](#histogram) with a cumulative aggregation temporali - `{name}_sum` metric denoting the sum field of the histogram, reported only if the sum is positive and monotonic. The sum is positive and monotonic when all buckets are positive. All attributes of the histogram point are converted to Prometheus labels. - A series of `{name}` metric points that contain all attributes of the histogram point recorded as labels. Additionally, a label, denoted as `le` is added denoting the bucket boundary. The label's value is the stringified floating point value of bucket boundaries, ordered from lowest to highest. The value of each point is the sum of the count of all histogram buckets up the the boundary reported in the `le` label. These points will include a single exemplar that falls within `le` label and no other `le` labelled point. The final bucket metric MUST have an `+Inf` threshold. -OpenTelemetry Histograms with Delta aggregation temporality SHOULD be aggregated into a Cumulative aggregation temporality and follow the logic above, or MAY be dropped. +OpenTelemetry Histograms with Delta aggregation temporality SHOULD be aggregated into a Cumulative aggregation temporality and follow the logic above, or MUST be dropped. #### Summaries From bd4ca10f7abf1a1824a8f6f53f9317bfb94ebe65 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 9 Feb 2022 12:20:46 -0800 Subject: [PATCH 21/26] allow process_start_time_seconds, but specify that it must be disabled by default --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index bc49538901f..0180e2e54dd 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1085,7 +1085,7 @@ The following Prometheus types MUST be dropped: #### Start Time -Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion MUST follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time). +Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion MUST follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time) by default. Conversion MAY offer configuration, disabled by default, which allows using the `process_start_time_seconds` metric to provide the start time. #### Exemplars From 8842df54ab6126820c7dbb54816982426a18de3d Mon Sep 17 00:00:00 2001 From: Anuraag Agrawal Date: Thu, 10 Feb 2022 09:41:56 +0900 Subject: [PATCH 22/26] Update specification/metrics/datamodel.md Co-authored-by: Tyler Yahn --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 0180e2e54dd..87e315ce8fa 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1149,7 +1149,7 @@ An [OpenTelemetry Summary](#summary-legacy) MUST be converted to a Prometheus me #### Dropped Data Points -The following OTLP data points must be dropped: +The following OTLP data points MUST be dropped: * [ExponentialHistogram](#exponentialhistogram) From 4303b79f9819895bfb0d2b958d49ada743b43486 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 10 Feb 2022 09:17:52 -0500 Subject: [PATCH 23/26] Update specification/metrics/datamodel.md Co-authored-by: legendecas --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 87e315ce8fa..e413cdb95e4 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1163,7 +1163,7 @@ OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https:/ #### Resource Attributes -In SDK Prometheus (pull) exporters, all resource attributes MUST be dropped, and MUST NOT be not attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping. +In SDK Prometheus (pull) exporters, all resource attributes MUST be dropped, and MUST NOT be attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping. In the Collector's Prometheus pull and push (remote-write) exporters, it is possible for metrics from multiple targets to be sent together, so targets must be disambiguated from one another. However, the Prometheus exposition format and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto) formats do not include a notion of resource, and expect metric labels to distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances) labels distinguish targets and are expected to be present on metrics exposed on a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/) Prometheus endpoint) or pushed via Prometheus remote-write. In the collector Prometheus exporters, the `job` and `instance` resource attributes MUST be converted to Prometheus metric labels, and other resource attributes SHOULD NOT be converted to metric labels. From 1ea331550b9f2968e99b4d296c42f71729a8bc94 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 11 Feb 2022 13:45:01 -0500 Subject: [PATCH 24/26] Apply suggestions from code review Co-authored-by: Joshua MacDonald --- specification/metrics/datamodel.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index e413cdb95e4..bec0cad0c5d 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1045,7 +1045,7 @@ OpenTelemetry metric data. Since OpenMetrics has a superset of Prometheus' types #### Counters -A [Prometheus Counter](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#counter) MUST be converted to an OTLP Sum. +A [Prometheus Counter](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#counter) MUST be converted to an OTLP Sum with `is_monotonic` equal to `true`. #### Gauges @@ -1061,7 +1061,7 @@ A [Prometheus Histogram](https://github.com/OpenObservability/OpenMetrics/blob/m Multiple Prometheus histogram metrics MUST be merged together into a single OTLP Histogram: -* The `le` label on non-suffixed metrics is used to identify histogram bucket boundaries. Each Prometheus line produces one bucket on the resulting histogram. +* The `le` label on non-suffixed metrics is used to identify and order histogram bucket boundaries. Each Prometheus line produces one bucket count on the resulting histogram. Each value for the `le` label except `+Inf` produces one bucket boundary. * Lines with `_count` and `_sum` suffixes are used to determine the histogram's count and sum. * If `_count` is not present, the metric MUST be dropped. * If `_sum` is not present, it MUST be computed from the buckets. @@ -1116,7 +1116,7 @@ An [OpenTelemetry Gauge](#gauge) MUST be converted to a Prometheus Gauge. [OpenTelemetry Sums](#sums) follows this logic: -- If the aggregation temporality is cumulative and the sum is monotonic, it MUST be converted to a Prometheus Sum. +- If the aggregation temporality is cumulative and the sum is monotonic, it MUST be converted to a Prometheus Counter. - If the aggregation temporality is cumulative and the sum is non-monotonic, it MUST be converted to a Prometheus Gauge. - If the aggregation temporality is delta and the sum is monotonic, it SHOULD be converted to a cumulative temporality and become a Prometheus Sum - Otherwise, it MUST be dropped. From 9aa19f513dabead0938ce3ebdad36d36cd4b4ac3 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 11 Feb 2022 11:18:18 -0800 Subject: [PATCH 25/26] describe when process_start_time_seconds isn't correct --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index bec0cad0c5d..3552523f01b 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1085,7 +1085,7 @@ The following Prometheus types MUST be dropped: #### Start Time -Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion MUST follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time) by default. Conversion MAY offer configuration, disabled by default, which allows using the `process_start_time_seconds` metric to provide the start time. +Prometheus Cumulative metrics do not include the start time of the metric. When converting Prometheus Counters to OTLP, conversion MUST follow [Cumulative streams: handling unknown start time](#cumulative-streams-handling-unknown-start-time) by default. Conversion MAY offer configuration, disabled by default, which allows using the `process_start_time_seconds` metric to provide the start time. Using `process_start_time_seconds` is only correct when all counters on the target start after the process and are not reset while the process is running. #### Exemplars From 7fa4164d5dbab8f4ca07887bde4b61fedf20d7d9 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 14 Feb 2022 08:02:05 -0800 Subject: [PATCH 26/26] convert values to string --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 3552523f01b..879cde2ce2c 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1155,7 +1155,7 @@ The following OTLP data points MUST be dropped: #### Metric Attributes -OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). String Attribute values are converted directly to Metric Attributes, and non-string Attribute values SHOULD be replaced with an empty string. Prometheus metric label keys are required to match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names MUST replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same location. In such unlikely cases, if multiple key-value pairs are converted to have the same Prometheus key, the values MUST be concatenated together, separated by `;`, and ordered by the lexicographical order of the original keys. +OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https://Prometheus.io/docs/concepts/data_model/#metric-names-and-labels). String Attribute values are converted directly to Metric Attributes, and non-string Attribute values MUST be converted to string attributes following the [attribute specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/common.md#attribute). Prometheus metric label keys are required to match the following regex: `[a-zA-Z_:]([a-zA-Z0-9_:])*`. Metrics from OpenTelemetry with unsupported Attribute names MUST replace invalid characters with the `_` character. This may cause ambiguity in scenarios where multiple similar-named attributes share invalid characters at the same location. In such unlikely cases, if multiple key-value pairs are converted to have the same Prometheus key, the values MUST be concatenated together, separated by `;`, and ordered by the lexicographical order of the original keys. #### Exemplars