Add user label to ingested samples metrics

[aknuds1]

What this PR does

Add user label to ingested samples metrics:

• cortex_ingester_ingested_samples_total
• cortex_ingester_ingested_samples_failures_total

The motivation is that we need to be able to differentiate on the tenant when determining the percentage of samples getting dropped by our ingesters.

TODO:

• Introduce recording rule for use from alerts
• Determine if these changes are OK to introduce in Mimir 2.1.0 (i.e., are they breaking changes?)

Which issue(s) this PR fixes or relates to

Fixes #1531.

Checklist

• Tests updated
• [na] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

cortex_ingester_ingested_samples_total

This metric is used in several alerts. Adding a per-user label it will become high cardinality, and so such alerts more complex to evaluate. We should check if for some of such alerts we can use a different metric, or discuss alternative ideas (eg. add recording rule).

[pracucci]

Changes LGTM. Let's look at where these metrics are used and then we can merge.

[aknuds1]

Note to self:

Marco suggests adding recording rule named cluster_job_instance:cortex_ingester_ingested_samples_total:rate1m:

sum by(cluster, job, instance) (rate(cortex_ingester_ingested_samples_total[1m]))

[aknuds1]

@pracucci with the help of @pstibrany I've crafted in total three rules, and converted relevant alerts to use them. PTAL when you have the time :)

[CLAassistant]

All committers have signed the CLA.

[pstibrany]

Please rebase this PR on top of main to make all required test checks pass.

[pracucci]

Good job! The recording and alerting rules LGTM 👏

There's just a thing I would like you to check: why the compiled mixin .yaml files have been reformatted? I can't explain it. Could you try to change the make build-mixin target to run in the build-image (needs to be moved in the Makefile) and check if they're still reformatted?

[pracucci]

There's just a thing I would like you to check: why the compiled mixin .yaml files have been reformatted? I can't explain it. Could you try to change the make build-mixin target to run in the build-image (needs to be moved in the Makefile) and check if they're still reformatted?

I answered my question (and fixed Makefile) in this PR:
#1606

Please rebase once merged. Thanks!

[pracucci]

LGTM, thanks!