Distributor: Use pooled buffers for reading/decompressing request body

[aknuds1]

What this PR does

The distributor currently uses pooled buffers for decompressing snappy compressed remote write requests. However, it doesn't use pooled buffers when reading/gzip decompressing (OTLP requests can be gzip compressed) request bodies. In this PR I propose to use buffer pooling also for reading uncompressed/compressed write request bodies, which should benefit OTLP requests as they're not snappy compressed.

Benchmarks

Benchmarking shows some speedup and reduction in memory usage for OTLP. I'm currently testing these changes in a dev cell, with no problems detected.

BenchmarkPushHandler:

✗ benchstat push.txt push-buffer.txt
goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/distributor
cpu: AMD Ryzen 9 3950X 16-Core Processor            
               │  push.txt   │          push-buffer.txt           │
               │   sec/op    │   sec/op     vs base               │
PushHandler-32   2.926µ ± 2%   2.976µ ± 2%  +1.73% (p=0.007 n=10)

               │  push.txt  │       push-buffer.txt        │
               │    B/op    │    B/op     vs base          │
PushHandler-32   638.5 ± 1%   636.0 ± 1%  ~ (p=0.323 n=10)

               │  push.txt  │        push-buffer.txt         │
               │ allocs/op  │ allocs/op   vs base            │
PushHandler-32   16.00 ± 0%   16.00 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

BenchmarkOTLPHandler:

✗ benchstat otlp.txt otlp-buffer.txt
goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/distributor
cpu: AMD Ryzen 9 3950X 16-Core Processor            
                        │   otlp.txt   │          otlp-buffer.txt          │
                        │    sec/op    │   sec/op     vs base              │
OTLPHandler/protobuf-32   1017.5µ ± 1%   991.8µ ± 9%  -2.52% (p=0.002 n=6)
OTLPHandler/JSON-32        1.409m ± 2%   1.386m ± 5%  -1.65% (p=0.015 n=6)
geomean                    1.198m        1.173m       -2.09%

                        │   otlp.txt   │           otlp-buffer.txt           │
                        │     B/op     │     B/op      vs base               │
OTLPHandler/protobuf-32   471.5Ki ± 0%   450.1Ki ± 0%   -4.53% (p=0.002 n=6)
OTLPHandler/JSON-32       553.9Ki ± 0%   497.6Ki ± 0%  -10.15% (p=0.002 n=6)
geomean                   511.0Ki        473.3Ki        -7.38%

                        │  otlp.txt   │          otlp-buffer.txt          │
                        │  allocs/op  │  allocs/op   vs base              │
OTLPHandler/protobuf-32   7.136k ± 0%   7.135k ± 0%  -0.01% (p=0.002 n=6)
OTLPHandler/JSON-32       10.17k ± 0%   10.17k ± 0%  -0.01% (p=0.002 n=6)
geomean                   8.519k        8.518k       -0.01%

Which issue(s) this PR fixes or relates to

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[pstibrany]

Nice. I'm not sure if tying the two buffers (body / decoded) into single struct is a good idea, and I'm wondering if we should use a struct that would track all buffers used by request. Something like:

type requestBuffers struct {
	p       *sync.Pool
	buffersFromThePool []*bytes.Buffer
}

func (rb *requestBuffers) getBuffer() *bytes.Buffer {
	b := rb.p.Get().(*bytes.Buffer)
	rb.buffersFromThePool = append(rb.buffersFromThePool, b)
	return b
}

func (rb *requestBuffers) cleanup() {
	for _, b := range rb.buffersFromThePool {
		rb.p.Put(b)
	}
	rb.buffersFromThePool = rb.buffersFromThePool[:0]
}

Then we could call cleanup to return all buffers.

[aknuds1]

Thanks for the advice @pstibrany - I think I agree it could be beneficial to avoid coupling the two buffers together in the pool, so I implemented your suggested design instead (type util.RequestBuffers). PTAL.

The revised design fares slightly worse in the distributor push benchmark, with one extra allocation, but I guess it makes little practical difference?

✗ benchstat push.txt push-request.txt
goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/distributor
cpu: AMD Ryzen 9 3950X 16-Core Processor            
               │  push.txt   │       push-request.txt        │
               │   sec/op    │   sec/op     vs base          │
PushHandler-32   27.09µ ± 2%   27.49µ ± 3%  ~ (p=0.247 n=10)

               │  push.txt  │          push-request.txt          │
               │    B/op    │    B/op     vs base                │
PushHandler-32   633.5 ± 1%   744.5 ± 1%  +17.52% (p=0.000 n=10)

               │  push.txt  │         push-request.txt          │
               │ allocs/op  │ allocs/op   vs base               │
PushHandler-32   16.00 ± 0%   17.00 ± 0%  +6.25% (p=0.000 n=10)

[pstibrany]

Thank you, changes make sense to me. I left some non-blocking suggestions.