Reduce stack allocations by reusing goroutines when pushing series from the distributors to the ingesters

[colega]

Is your feature request related to a problem? Please describe.

We've observed that around 5% of distributor CPU is spent calling runtime.newstack from the goroutines created by ring.DoBatch.

It would be nice to avoid those allocations.

Describe the solution you'd like

Ideally, this should be handled by the language itself, but it seems to be a recurring issue, ref: golang/go#18138

A workaround is to use a pool of goroutines to handle these calls, which would have their stacks reused.

Describe alternatives you've considered

Also considered growing the stack artificially using the growStack() method implementation from golang/go#18138

Steps

• Add support to ring.DoBatch PR
• Add -distributor.ingester-push-worker-goroutines flag to Mimir PR
• Experiment

[colega]

I deployed this to our ops environment, and I can see a 7.5% reduction in CPU per request.