OOM due to underestimated memory use in metadata handler

[travisdowns]

Version & Environment

Redpanda version: dev f75ceed

What went wrong?

Redpanda node continually runs out of memory at start when joining a loaded cluster. An underlying cause is that the size of metadata requests (and probably other types) is underestimated by several orders of magnitude in our memory throttling logic. This results in many requests executing in parallel, exhausting the per-shard memory.

The crashes are correlated with (a) startup or (b) another node being killed because in those cases the metadata requests pile up as we are trying to either (a) get the first metadata refresh or (b) figuring out who the new controller is, and many requests pile up on the refresh semaphore and then are suddenly uncorked all at once, causing a temporary and perhaps fatal spike in memory use.

What should have happened instead?

Our kafka request logic should throttle the requests to an an appropriate concurrent level to avoid OOM.

How to reproduce the issue?

1. Connect many consumers to a cluster.
2. Kill a node and restart it.
3. Observe (with added logging) the concurrency and memory usage for metadata requests.

Additional information

In reserve_request_units we calculate an estimated "memory size" of the request. This estimate is based on the over-the-wire size of the request: specifically we estimate we need request_size * 2 + 8000 bytes. In the case of metadata requests, this request is about 26 bytes on the wire, so we estimate that 8052 bytes are needed and "reserve" that many units from the memory semaphore.

However, the actual memory size to handle the request is not related to the on-the-wire size, but rather the size of the response. With many partitions, the response can be large: at least 100 bytes per partition just in "working space" alone, plus the size of the on-the-wire response which is similar. For 20k partitions that's probably in the range of 4 MB (we see memory allocation failures for single allocations in this range). So this is about 500x larger than the estimate and the memory semaphore provides no real protection (e.g., with 600M allocated to Kafka requests, we should let at most 150 requests run concurrently, but the existing throttle will let ~75,000 requests go in parallel).

It is not clear how best to fix this. A better estimate would solve the problem, but very early in the request handling flow it is not clear we can make a much better estimate: in this example we haven't built the topics table at all as we are starting up, so the requests are already in progress before we know there are many partitions. As a workaround, we can add a second semaphore inside the metadata handler: in this location we can make a better estimate of the size of the response (before we start building it) and limit the total concurrency to a response-size-aware value.