Add a limit on max remote write request samples

[aknuds1]

Is your feature request related to a problem? Please describe.

We need a limit on the maximum number of samples per remote write request (including OTLP). The motivation is that OTLP write requests are typically batched, and could contain so many samples that Mimir takes too long in processing them.

Describe the solution you'd like

A limit on the number of samples a remote write request is permitted to contain. When the limit is hit, the request should be rejected with HTTP 413. A suggested default is 10k.

Describe alternatives you've considered

Additional context

See https://github.com/grafana/mimir-squad/issues/2180, which mentions the following:

Another idea was to validate normal distributor limits prior to translating OTLP to Mimir requests, but this is much more difficult since the validation middlewares currently expect a normal Mimir write request.

[ying-jeanne]

Prior to enabling the default value of 10k, we intend to introduce a new metrics, potentially in the form of a histogram, to monitor the request samples value, so that give some time for collector configuration change. This approach ensures that no requests are rejected due to the updated default value, preventing any potential data loss.

[ying-jeanne]

The histograms on the client side do not provide the accurate total sample count; the actual series number should be the number of buckets plus 2, but the client side incorrectly counts this as 1. Configuring batch sizes appropriately on the client side poses a challenge.

[aknuds1]

According to Ying, these are classical histograms. Uncertain as to whether the same behaviour holds for native histograms.

[ying-jeanne]

related ticket #8260

[ying-jeanne]

Prior to enabling the default value of 10k, we intend to introduce a new metrics, potentially in the form of a histogram, to monitor the request samples value, so that give some time for collector configuration change. This approach ensures that no requests are rejected due to the updated default value, preventing any potential data loss.

this is implemented.