-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mimir rejects valid exemplars with err-mimir-exemplar-labels-missing
#7904
Comments
The Prometheus TSDB code drops exemplars with no labels which is why Mimir rejects them on ingest. It wasn't a problem in practice until now because Prometheus never sends exemplars with no labels. |
Discussion here when some of the logic to reject empty exemplars was added: #873 |
@gouthamve since this behaviour is by design in Prometheus, can the issue be closed? Or do you think it should be reconsidered in Prom? |
I think it needs to be reconsidered in Prometheus. We should atleast not drop them with 400s because OTel seems produce a ton of these empty exemplars. When that happens, the customers alerts start firing because their writes are failing, and the logs are full of the |
I'm curious, what's the use case for an exemplar without labels? |
👍 yes, I think it's just example values. As it's defined in the OpenMetrics spec, and in the OpenTelemetry spec, we should support this. It's maybe ok to ignore the Exemplar itself, but dropping the entire time series is certainly not spec compliant. |
I think that dropping the entire time series is a critical bug that deserves a separate issue. I also think that we should not reject the empty exemplars just because Prometheus doesn't send those. |
err-mimir-exemplar-labels-missing
When visualized, the values give a better idea of distribution. Especially when buckets are coarse, exemplars can suggest you have a particular modal value, or bimodal distribution, etc. |
Thank you, @bboreham, I figured out that myself :D |
I created a feature request upstream prometheus/prometheus#14208 |
I believe this was a misunderstanding. Nick's source is me at #873, where I said it would be confusing. |
Closed in #8224 |
Describe the bug
Seeing errors like:
From: open-telemetry/opentelemetry-go-contrib#5383 (comment)
To Reproduce
Send an exemplar with no labels to Grafana Cloud.
Expected behavior
Return a 200 OK, not a 400 error.
The text was updated successfully, but these errors were encountered: