Metrics cardinality reduction

[dotnwat]

PRD: https://docs.google.com/document/d/13q3BgBxD518hLqBooubTw4RYu3M6d6F22o05XuVedCU/edit
RFC: https://docs.google.com/document/d/1TxaJiNIyMAHr3cNpebd9K4wylih4jviFqv5-JuR-FDw/edit

Scope:

• Create a new prometheus metrics endpoint including the metrics requested in the PRD and having a much lower cardinality than the old endpoint (no per-shard metrics unless we actively want that for particular metrics, and minimal per-partition or per-topic metrics).
• Document all metrics that are included in the new metrics endpoint
• Make old endpoint optional, but enabled by default for backward compatibility.
• Test at scale, e.g. a cluster with 10,000 partitions. Consider extending ManyPartitionsTest to include a rough check on the metrics count, so that if we accidentally increase it later a test will fail.

Out of scope:

• Any output format other than prometheus
• Any optimization or other improvement of the existing metrics endpoint
• Any configurability of what's included in the new metrics endpoint (it's all or nothing).

[BenPope]

Some PoC code for configuration and aggregation of existing metrics: #4843

[BenPope]

Hey team! Please add your planning poker estimate with ZenHub @BenPope @NyaliaLui

[BenPope]

This has been replaced by a different mechanism.