Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

receiver/prometheus: emit staleness markers on missing metrics after scrapes #3413

Closed
odeke-em opened this issue Jun 10, 2021 · 1 comment · Fixed by #3423
Closed

receiver/prometheus: emit staleness markers on missing metrics after scrapes #3413

odeke-em opened this issue Jun 10, 2021 · 1 comment · Fixed by #3423
Labels
area:exporter bug Something isn't working

Comments

@odeke-em
Copy link
Member

Coming here from #2961 and then from the Prometheus RemoteWrite exporter test failures, which set up a Prometheus receiver and then request a scrape then transmit the retrieved scrapes to the pipeline then eventually to the remote write endpoint. When metrics disappear, we need to emit staleness markers aka a special NaN per https://github.com/prometheus/prometheus/blob/39d79c3cfb86c47d6bc06a9e9317af582f1833bb/pkg/value/value.go#L28 of value whose bits from float64 have a value of StaleNaN uint64 = 0x7ff0000000000002
@odeke-em odeke-em added the bug Something isn't working label Jun 10, 2021
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 10, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
72934eb 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
@odeke-em odeke-em mentioned this issue Jun 10, 2021
Merged
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 10, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
04ac4d6 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 10, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
ef1d34d 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
@odeke-em odeke-em mentioned this issue Jun 11, 2021
Merged
@odeke-em
Copy link
Member Author

The PR to finish things is #3423 but unfortunately Github doesn't stack/link changes so that PR has code from this one, but after this one is merged then it'll get to its normal size.

odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 11, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
7c6b172 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 14, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
658d42d 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 15, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics
eb85405 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates open-telemetry#3413
Updates open-telemetry#2961
tigrannajaryan pushed a commit that referenced this issue Jun 16, 2021
@odeke-em
receiver/prometheus: add store to track stale metrics (#3414)
cdc1634 
Implements a simple store that will be used to record stale metrics;
those that are NOT present in the current scrape, but were in the prior
scrape. This logic will then be attached to our receiver logic
to then complete the mechanism and then issue stale markers.

Updates #3413
Updates #2961
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 16, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
1a1339f 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 17, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
6ab3223 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 17, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
4e062a8 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jun 22, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
cc40e65 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jul 6, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
8fa960f 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jul 8, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
5a16fd7 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
odeke-em added a commit to orijtech/opentelemetry-collector that referenced this issue Jul 8, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
572bb06 
…red metrics

Adds the staleness store to the metricsBuilder and transaction,
to track when metrics disappear between scrapes.

Fixes open-telemetry#2961
Fixes open-telemetry#3413
tigrannajaryan pushed a commit that referenced this issue Jul 8, 2021
@odeke-em
receiver/prometheus: glue and complete staleness marking for disappea…
8b79380 
…ring metrics (#3423)

Adds the staleness store to the metricsBuilder and transaction to track when metrics
disappear between scrapes.

Depends on #3414
Fixes #3413
Fixes #2961
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:exporter bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@alolita @odeke-em