Querier Read Limits

[bwplotka]

With more streamed fanout (#5296) we are finally able to add more construction series / sample limits to Querier for QoS. This is because we know immediately how many series and samples globally we fetch into Querier instead of learning about that only after we computed most of it.

NOTE: This does not mean we don't need limits on other components like Store (there is already some limit there):

ref

However, we can control a lot from the Querier when it comes to QoS. The main problem comes from the fact that queries are never uniform. Some are ultra-small (samples over last 5 minutes, 10 series), some ultra large, (samples over year, 100 millions series). This why only only "limit" options we have on Querier are not enough:

ref

With query-frontend we can limit this problem to splitting queries to a 1d time window, but this has still some downsides:

• The same vertical cardinality problem appears (e.g. millions of series with one sample at the same time)
• We can't leverage downsampled data with such approach.

AC:

• Short term:
  • Ability to add query limits so w
• Long:
  • Limits per tenant
  • Dynamic per query limit, based on available capacity in Querier (other queries).

Initial Ideas:

• Start with a simple approach first - add series/sample limit to Qurier.
• Find metric that will represent the limiting factor (both mem and latency/CPU) e.g bytes processed or perhaps similar to write DPM

cc @GiedriusS

[douglascamata]

I am also very interested in this.

Is there already a configurable timeout for queries that will cause work cancellation to be pushed downstream?

[douglascamata]

• Long:
  • Limits per tenant

So there is some interest in making other parts of Thanos tenant-aware via configuration of a tenant label name?