[vtgate] Add flag to pool connection read buffers

[brirams]

Description

We(Slack) found that when a vtgate is under heavy connection load, tail latencies of connect times
tend to spike into the tens of milliseconds. A contributing factor is GC slowdown that is
exacerbated by the repeated allocation of enw 16k read buffers for each incoming connection. This PR
adds a flag to the vtgate mysql server to enable pooling those read buffers and re-using them across
requests.

@henryr found that under a synthetic load, we were able to get connection times down by 55% at the
p999 and 40% at the p99:

We've also run this against our loadtest environment and found, aside from less GC pauses, that our
upstream clients saw a 20% reduction in connection failures, with a 5-8% drop in connect latencies
across the board(p50 -> p999), and no perceived degradation in query latency.

This type of change has been attempted in the past
and was closed out but since there has since been work to buffer connection writes, figured it made
sense to revisit that.

Related Issue(s)

#4186

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

n/a

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
• If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive. Additionally, flag names should use dashes (-) as word separators rather than underscores (_).
• If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.

[harshit-gangal]

can we run the benchmarks from here go/mysql/query_benchmark_test.go if it gives some additional insights?

[brirams]

Benchmark output from this branch(still need to compare with main but figured this was good to have for now):

 ~/n/d/vitess/go/mysql $ go test -v -bench=.
<elided>
goos: darwin
goarch: amd64
pkg: vitess.io/vitess/go/mysql
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkParallelShortQueries
BenchmarkParallelShortQueries-16     	       1	1428472710 ns/op	   0.00 MB/s
BenchmarkParallelMediumQueries
BenchmarkParallelMediumQueries-16    	       1	1482719290 ns/op	   0.01 MB/s
BenchmarkParallelRandomQueries
BenchmarkParallelRandomQueries-16    	       1	1467065363 ns/op
PASS

[brirams]

results off of the main branch:

~/n/d/vitess/go/mysql $ go test -v -bench=.
<elided>
goos: darwin
goarch: amd64
pkg: vitess.io/vitess/go/mysql
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkParallelShortQueries
BenchmarkParallelShortQueries-16     	       1	1588925544 ns/op	   0.00 MB/s
BenchmarkParallelMediumQueries
BenchmarkParallelMediumQueries-16    	       1	1576328375 ns/op	   0.01 MB/s
BenchmarkParallelRandomQueries
BenchmarkParallelRandomQueries-16    	       1	1627565065 ns/op
PASS

[brirams]

modified the benchmark to run with the read buffer pooling enable and disabled so we can compare more effectively:

~/n/d/vitess/go/mysql $ go test -v -bench=.
<elided>
goos: darwin
goarch: amd64
pkg: vitess.io/vitess/go/mysql
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkParallelShortQueries
BenchmarkParallelShortQueries-16                          	       1	1820589019 ns/op	   0.00 MB/s
BenchmarkParallelMediumQueries
BenchmarkParallelMediumQueries-16                         	       1	1798463863 ns/op	   0.01 MB/s
BenchmarkParallelRandomQueries
BenchmarkParallelRandomQueries-16                         	       1	1712101024 ns/op
BenchmarkParallelShortQueriesWithReadBufferPooling
BenchmarkParallelShortQueriesWithReadBufferPooling-16     	       1	1687282698 ns/op	   0.00 MB/s
BenchmarkParallelMediumQueriesWithReadBufferPooling
BenchmarkParallelMediumQueriesWithReadBufferPooling-16    	       1	1740339458 ns/op	   0.01 MB/s
BenchmarkParallelRandomQueriesWithReadBufferPooling
BenchmarkParallelRandomQueriesWithReadBufferPooling-16    	       1	1702365597 ns/op
PASS

[harshit-gangal]

changes looks good, just one flag naming convention change.

[harshit-gangal]

Would need to add in release notes for the new flag and what it means over https://github.com/vitessio/vitess/blob/main/doc/releasenotes/15_0_0_summary.md