Query-scheduler performance issue after #5880

[pracucci]

We're analyzing some read path latency which recently got worse and we've found that looks like there's an issue with the query-scheduler after #5880, which causes the time it takes to enqueue a query in the query-scheduler to grow over time. Looking when the issue started, it began with the rollout of the weekly release n. 255.

The average enqueue latency is measured using:

sum by(namespace) (rate(cortex_query_scheduler_enqueue_duration_seconds_sum{container="query-scheduler"}[5m]))
/
sum by(namespace) (rate(cortex_query_scheduler_enqueue_duration_seconds_count{container="query-scheduler"}[5m]))
* 1000

We can observe that it grows over time. For example, this is our staging environment:

These are a couple of our production environments:

[pracucci]

In our staging environment, it looks like the issue didn't show after upgrading from weekly release 255 to 256. The weekly release 256 was rolled out on 2023-09-20 at 16:30 UTC:

[charleskorn]

I believe #6100 will fix this issue. See the description in the PR for more details.