Consider removing blocks-storage.bucket-store.max-chunk-pool-bytes limits

[grafanabot]

Is your feature request related to a problem? Please describe.
blocks-storage.bucket-store.max-chunk-pool-bytes is used to configure the maximum size of the chunk pool in the store-gateway bucket store. The chunk byte pool is used to reduce allocations but having the limit enforced can cause queries to fail when the pool is exhausted. I believe the purpose of the limit being enforced is to prevent store-gateway OOM kills but since you end up needing to set the limit to a much higher value to satisfy queries (e.g. 80% of the store-gateway requested memory) you don't really gain any extra protection.

Describe the solution you'd like
Removal of the blocks-storage.bucket-store.max-chunk-pool-bytes flag and associated configuration and using a chunk pool with no maximum.
Submitted by: jdbaldry
Cortex Issue Number: 3793

[grafanabot]

I'm also dubious about this limit at all. We use memory pools in other parts of the code base and they're never limited. Unless I'm missing something, this is the only pool we limit.

Submitted by: pracucci

[grafanabot]

The PR cortexproject/cortex#3830 improved it a bit. Before the chunks pool was per-tenant, after it's global (shared across all tenants).

Submitted by: pracucci

[grafanabot]

I keep having mixed feelings about this. Issue still valid.

Submitted by: pracucci

[dimitarvdimitrov]

resolved by #4996