Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Exemplars for http_request_duration_seconds histogram #3977

Merged
merged 1 commit into from
Mar 26, 2021

Conversation

metalmatze
Copy link
Contributor

@metalmatze metalmatze commented Mar 26, 2021

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

The Prometheus HTTP handler now enables OpenMetrics as response format upon content negotiation.
You can test it by running curl -s -H 'Accept: application/openmetrics-text' localhost:10904/metrics
This allows us to pass a traceID to ObserveWithExemplar.

A lot of the other changes are actually changing the order of calling the middlewares.
Going forward we need to make sure to first call the tracing middleware so that by the time our monitoring/instrumentation middleware gets hit the tracing spans are already populated.

Verification

Tested locally with Jaeger and Prometheus running --enable-feature=exemplar-storage.

Screenshot from 2021-03-26 12-47-25

Closes #3944

@metalmatze metalmatze requested a review from yeya24 March 26, 2021 11:49
Signed-off-by: Matthias Loibl <mail@matthiasloibl.com>

A lot of these changes are actually changing the order of calling the
middlwares. Going forward we need to make sure to first call the tracing
middleware so that by the time our monitoring/instrumentation middleware
gets hit the tracing spans are already populated.
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sooo good. Thanks @metalmatze !

pkg/extprom/http/instrument_server.go Show resolved Hide resolved
@bwplotka bwplotka merged commit 3016237 into thanos-io:main Mar 26, 2021
@metalmatze metalmatze deleted the exemplars branch March 26, 2021 12:35
// If we find a tracingID we'll expose it as Exemplar.
span := opentracing.SpanFromContext(r.Context())
if span != nil {
spanCtx, ok := span.Context().(jaeger.SpanContext)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we just support Jaeger tracer? Is it compatible with lightstep, stackdriver, etc?

if span != nil {
spanCtx, ok := span.Context().(jaeger.SpanContext)
if ok {
observer.(prometheus.ExemplarObserver).ObserveWithExemplar(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bwplotka
Copy link
Member

Let's iterate over it @yeya24 good points. Also I would see GetTraceID as some helper function tbh (:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Instrument http_request_duration_seconds metric with exemplars
3 participants