Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meter name not in Prometheus scraper output? #4424

Closed
JamesNK opened this issue Apr 24, 2023 · 11 comments
Closed

Meter name not in Prometheus scraper output? #4424

JamesNK opened this issue Apr 24, 2023 · 11 comments
Labels
enhancement New feature or request pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package

Comments

@JamesNK
Copy link
Contributor

JamesNK commented Apr 24, 2023 •
edited
Loading

I'm testing out System.Diaganostics.Metrics with prometheus-net and opentelemetry-dotnet.

There is a difference in the metrics names in the Prometheus scraping endpoint:

  • prometheus-net includes the meter and counter name. For example, microsoft_aspnetcore_servers_kestrel_current_connections{endpoint="[::1]:7269"} 1
  • OTel only has the counter name. For example, current_connections{endpoint="[::1]:7269"} 1

It seems wrong that these are inconsistent. Grafana dashboards expecting names that include the meter name will fail if it's missing, or vice versa.

The prometheus-net approach is more verbose, but it won't fail if counters with the same name are created by different meters? For example, ASP.NET Core is going to have current_connections in two meters: Microsoft.AspNetCore.Servers.Kestrel and Microsoft.AspNetCore.Http.Connections.
@JamesNK JamesNK added the question Further information is requested label Apr 24, 2023
@pellared
Copy link
Member

As far as I know the meter name should be set as otel_scope_name label. Can you please double-check if it is present?

Reference to the OTel specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#instrumentation-scope

@JamesNK
Copy link
Contributor Author

JamesNK commented Apr 24, 2023

My situation is neither OTel converted to Prometheus or Prometheus converted to OTel.

These are custom meters and counters created using System.Diagnostics.Metrics.

@pellared
Copy link
Member

Are you not exporting the metrics using an Prometheus exporter (e.g. like this one)?

I added wrong hyperlink. It should be https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#instrumentation-scope-1 (same header just further in docs).

@JamesNK
Copy link
Contributor Author

JamesNK commented Apr 24, 2023 •
edited
Loading

Yes, I'm using the Prometheus scraping endpoint in that package.

This is the output when I visit /metrics:

# TYPE current_connections gauge
# HELP current_connections Number of connections that are currently active on the server.
current_connections{endpoint="[::1]:7269"} 1 1682342647119
current_connections{endpoint="127.0.0.1:7269"} 0 1682342647119
current_connections{endpoint="127.0.0.1:5072"} 1 1682342647119

# TYPE connection_duration_s histogram
# UNIT connection_duration_s s
# HELP connection_duration_s The duration of connections on the server.
connection_duration_s_bucket{endpoint="[::1]:7269",le="0"} 0 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="5"} 1 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="10"} 1 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="25"} 1 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="50"} 1 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="75"} 1 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="100"} 1 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="250"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="500"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="750"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="1000"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="2500"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="5000"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="7500"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="10000"} 4 1682342647119
connection_duration_s_bucket{endpoint="[::1]:7269",le="+Inf"} 4 1682342647119
connection_duration_s_sum{endpoint="[::1]:7269"} 550.8404837 1682342647119
connection_duration_s_count{endpoint="[::1]:7269"} 4 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="0"} 0 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="5"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="10"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="25"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="50"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="75"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="100"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="250"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="500"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="750"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="1000"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="2500"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="5000"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="7500"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="10000"} 1 1682342647119
connection_duration_s_bucket{endpoint="127.0.0.1:7269",le="+Inf"} 1 1682342647119
connection_duration_s_sum{endpoint="127.0.0.1:7269"} 0.018707 1682342647119
connection_duration_s_count{endpoint="127.0.0.1:7269"} 1 1682342647119

# TYPE queued_connections gauge
# HELP queued_connections Number of connections that are currently queued and are waiting to start.
queued_connections{endpoint="127.0.0.1:7269"} 0 1682342647119
queued_connections{endpoint="[::1]:7269"} 0 1682342647119
queued_connections{endpoint="127.0.0.1:5072"} 0 1682342647119

# TYPE queued_requests gauge
# HELP queued_requests Number of HTTP requests on multiplexed connections (HTTP/2 and HTTP/3) that are currently queued and are waiting to start.
queued_requests{endpoint="[::1]:7269",version="HTTP/2"} 0 1682342647119

# TYPE tls_handshake_duration_s histogram
# UNIT tls_handshake_duration_s s
# HELP tls_handshake_duration_s The duration of TLS handshakes on the server.
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="0"} 0 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="5"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="10"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="25"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="50"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="75"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="100"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="250"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="500"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="750"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="1000"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="2500"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="5000"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="7500"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="10000"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException",le="+Inf"} 1 1682342647119
tls_handshake_duration_s_sum{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException"} 0.0067942 1682342647119
tls_handshake_duration_s_count{endpoint="127.0.0.1:7269",exception_name="System.IO.IOException"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="0"} 0 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="5"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="10"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="25"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="50"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="75"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="100"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="250"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="500"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="750"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="1000"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="2500"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="5000"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="7500"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="10000"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",exception_name="System.IO.IOException",le="+Inf"} 1 1682342647119
tls_handshake_duration_s_sum{endpoint="[::1]:7269",exception_name="System.IO.IOException"} 0.0067942 1682342647119
tls_handshake_duration_s_count{endpoint="[::1]:7269",exception_name="System.IO.IOException"} 1 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="0"} 0 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="5"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="10"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="25"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="50"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="75"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="100"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="250"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="500"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="750"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="1000"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="2500"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="5000"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="7500"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="10000"} 4 1682342647119
tls_handshake_duration_s_bucket{endpoint="[::1]:7269",protocol="Tls13",le="+Inf"} 4 1682342647119
tls_handshake_duration_s_sum{endpoint="[::1]:7269",protocol="Tls13"} 0.0380355 1682342647119
tls_handshake_duration_s_count{endpoint="[::1]:7269",protocol="Tls13"} 4 1682342647119

# TYPE current_tls_handshakes gauge
# HELP current_tls_handshakes Number of TLS handshakes that are currently in progress on the server.
current_tls_handshakes{endpoint="[::1]:7269"} 0 1682342647119
current_tls_handshakes{endpoint="127.0.0.1:7269"} 0 1682342647119

# TYPE current_requests gauge
# HELP current_requests Number of HTTP requests that are currently active on the server.
current_requests{host="localhost",method="GET",port="7269",scheme="https"} 1 1682342647119
current_requests{host="host.docker.internal",method="GET",port="5072",scheme="http"} 0 1682342647119

# TYPE request_duration_s histogram
# UNIT request_duration_s s
# HELP request_duration_s The duration of HTTP requests on the server.
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="0"} 0 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="5"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="10"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="25"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="50"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="75"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="100"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="250"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="500"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="750"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="1000"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="2500"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="5000"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="7500"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="10000"} 10 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200",le="+Inf"} 10 1682342647119
request_duration_s_sum{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200"} 0.13516590000000003 1682342647119
request_duration_s_count{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/",scheme="https",status_code="200"} 10 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="0"} 0 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="5"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="10"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="25"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="50"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="75"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="100"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="250"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="500"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="750"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="1000"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="2500"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="5000"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="7500"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="10000"} 36 1682342647119
request_duration_s_bucket{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200",le="+Inf"} 36 1682342647119
request_duration_s_sum{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200"} 0.047631700000000034 1682342647119
request_duration_s_count{host="host.docker.internal",method="GET",port="5072",protocol="HTTP/1.1",scheme="http",status_code="200"} 36 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="0"} 0 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="5"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="10"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="25"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="50"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="75"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="100"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="250"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="500"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="750"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="1000"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="2500"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="5000"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="7500"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="10000"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200",le="+Inf"} 37 1682342647119
request_duration_s_sum{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200"} 0.0324075 1682342647119
request_duration_s_count{host="localhost",method="GET",port="7269",protocol="HTTP/2",scheme="https",status_code="200"} 37 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="0"} 0 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="5"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="10"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="25"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="50"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="75"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="100"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="250"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="500"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="750"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="1000"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="2500"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="5000"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="7500"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="10000"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200",le="+Inf"} 2 1682342647119
request_duration_s_sum{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200"} 0.001114 1682342647119
request_duration_s_count{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test1",scheme="https",status_code="200"} 2 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="0"} 0 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="5"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="10"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="25"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="50"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="75"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="100"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="250"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="500"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="750"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="1000"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="2500"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="5000"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="7500"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="10000"} 1 1682342647119
request_duration_s_bucket{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200",le="+Inf"} 1 1682342647119
request_duration_s_sum{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200"} 0.0005038 1682342647119
request_duration_s_count{host="localhost",method="GET",port="7269",protocol="HTTP/2",route="/test2",scheme="https",status_code="200"} 1 1682342647119

# EOF

@pellared
Copy link
Member

pellared commented Apr 24, 2023 •
edited
Loading

I think it is a bug per https://github.com/open-telemetry/opentelemetry-specification/blob/773ee656f92c7f591f2fd9c38df82c264a15184d/specification/compatibility/prometheus_and_openmetrics.md?plain=1#L260-L262:

Prometheus exporters MUST add the scope name as the otel_scope_name label and
the scope version as the otel_scope_version label on all metric points by
default, based on the scope the original data point was nested in.

@pellared pellared added bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package labels Apr 24, 2023
@cijothomas
Copy link
Member

I think it is a bug per https://github.com/open-telemetry/opentelemetry-specification/blob/773ee656f92c7f591f2fd9c38df82c264a15184d/specification/compatibility/prometheus_and_openmetrics.md?plain=1#L260-L262:

Prometheus exporters MUST add the scope name as the otel_scope_name label and
the scope version as the otel_scope_version label on all metric points by
default, based on the scope the original data point was nested in.

This part of the spec is relatively new to the spec itself, and is not yet implemented. I'd mark this as enhancement to Prometheus exporter, not a bug.

@cijothomas
Copy link
Member

There is a difference in the metrics names in the Prometheus scraping endpoint:

prometheus-net includes the meter and counter name. For example, microsoft_aspnetcore_servers_kestrel_current_connections{endpoint="[::1]:7269"} 1
OTel only has the counter name. For example, current_connections{endpoint="[::1]:7269"} 1

I don't think prometheus-net and OTel Prometheus exporters will produce exact same output. OTel Exporters follow the OTel mapping (from OTLP -> OpenMetrics/Prometheus), and MeterName's are not prefixed to form the metric name in that spec.

@pellared pellared removed the bug Something isn't working label Apr 24, 2023
@cijothomas cijothomas added enhancement New feature or request and removed question Further information is requested labels May 9, 2023
@JamesNK JamesNK mentioned this issue May 20, 2023
Closed
6 tasks
@samsp-msft
Copy link

There is a difference in the metrics names in the Prometheus scraping endpoint:
prometheus-net includes the meter and counter name. For example, microsoft_aspnetcore_servers_kestrel_current_connections{endpoint="[::1]:7269"} 1
OTel only has the counter name. For example, current_connections{endpoint="[::1]:7269"} 1

I don't think prometheus-net and OTel Prometheus exporters will produce exact same output. OTel Exporters follow the OTel mapping (from OTLP -> OpenMetrics/Prometheus), and MeterName's are not prefixed to form the metric name in that spec.

I think part of the problem here is that OTEL thinks it's the only game in town, and assumes that everything exposed via it will follow its patterns. This is potentially a problem with scenarios like .NET where we are mapping existing technologies into OTEL.

Can there be an option (probably defaulting to on) to include the metric as part of the meter name in OTEL. Its only in cases like the OTEL instrumentation libraries where they are mapping to names in the OTEL spec that not including the prefix makes sense.

@cijothomas
Copy link
Member

Can there be an option (probably defaulting to on) to include the metric as part of the meter name in OTEL

See the OTel -> Prometheus mapping spec. We'd need to be adhering to that. If that mapping is not sufficient, we should raise an issue in the spec, as this is not just a .NET only issue, any other languages using prometheus clients are affected.

https://github.com/open-telemetry/opentelemetry-specification/blob/773ee656f92c7f591f2fd9c38df82c264a15184d/specification/compatibility/prometheus_and_openmetrics.md?plain=1#L260-L262

@robertcoltheart
Copy link
Contributor

robertcoltheart commented Nov 24, 2023 •
edited
Loading

Can we revisit this issue? I noticed that both the OTEL exporter and the Prometheus exporter don't add otel_scope_name (in this case, the name of the meter) to the metric. Additionally, global tags that can be set in a meter now in .NET 8 are not exported either. I'm happy to raise a PR to add these in, just wanted to see if anyone is already addressing this?

Edit: Just noticed this issue, is this the same thing? #4563

@JamesNK
Copy link
Contributor Author

JamesNK commented Nov 25, 2023

There is an issue for adding otel_scope_name here: #3972

I'll close this issue as the question has been answered.

@JamesNK JamesNK closed this as not planned Won't fix, can't repro, duplicate, stale Nov 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@JamesNK @pellared @cijothomas @robertcoltheart @samsp-msft