-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Per-handler memory estimation, more accurate estimate for metadata handler #5346
Commits on Jul 14, 2022
-
Clarify behavior of process_next_response
The behavior of process_next_response is worth clarifying as the returned future does not nececessarily wait for all enqueued respones to be finished before resolving. We also rename the method to better reflect its purpose.
Configuration menu - View commit details
-
Copy full SHA for eea0de5 - Browse repository at this point
Copy the full SHA eea0de5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5f5c7c6 - Browse repository at this point
Copy the full SHA 5f5c7c6View commit details -
Release resources after the response is written
Currently we release resources after the response is enqueued in connection_context and response processing is called, but it may not have been sent at this point as we require in-order responses but second-stage processing may happen out of order. In this change, we instead tunnel the resource object through to the place where the response is written, and release it there. FIxes redpanda-data#5278.
Configuration menu - View commit details
-
Copy full SHA for d2bac6c - Browse repository at this point
Copy the full SHA d2bac6cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5def3b7 - Browse repository at this point
Copy the full SHA 5def3b7View commit details -
Move max_api_key function to handlers header.
This lets us share it with the request processing code which would also like to do type list based manipulation of the request types.
Configuration menu - View commit details
-
Copy full SHA for 79fe3a5 - Browse repository at this point
Copy the full SHA 79fe3a5View commit details -
Introduce KafkaApiHandlerAny concept
We already had concepts for one-phase and two-phase handlers, and this concept is simply the union of those two handler concepts, i.e., "any" type of handler.
Configuration menu - View commit details
-
Copy full SHA for 2e4dd22 - Browse repository at this point
Copy the full SHA 2e4dd22View commit details -
Rename handler to single_stage_handler
We wish to reclaim the name handler for the generic handler interface introduced in the next change.
Configuration menu - View commit details
-
Copy full SHA for 0c8a6e9 - Browse repository at this point
Copy the full SHA 0c8a6e9View commit details -
Introduce type-erased handler interface
This is a polymorphic handler class (abstract class with virtual methods), as well as concrete instantiations of the interface for each of the existing handlers, which can be looked up by API key. This lets us treat handlers generically without resorting to template functions which must be specialized for each handler type. This reduces code bloat significantly as we do not duplicate code paths for our ~45 handler types. For example, requests.cc.o drops from ~11 MB to ~5 MB after it is switched to the any_handler approach.
Configuration menu - View commit details
-
Copy full SHA for d533df6 - Browse repository at this point
Copy the full SHA d533df6View commit details -
Add handler_interface unit tests
The handler_interface already gets good functional coverage as it is added to the core request path in requests.cc, but we also include this unit test with basic coverage.
Configuration menu - View commit details
-
Copy full SHA for f80e99d - Browse repository at this point
Copy the full SHA f80e99dView commit details -
Update request handling to use polymorphic handler
Preceding changes in this series introduced a runtime polymorphic handler class, and this change switches most of the request handling to use it. In particular, we replace the large switch on API key which dispatches to a template method to a lookup of the handler method and virtual dispatch. Some handlers that need special processing like the authentication/SASL related ones still use the old approach for now.
Configuration menu - View commit details
-
Copy full SHA for 12d4379 - Browse repository at this point
Copy the full SHA 12d4379View commit details -
Add support for memory estimation to handlers
Currently we use single memory estimate for all kafka request types, but different API calls may use wildly different amounts of memory. This change allows each handler to perform an API-specific calculation instead.
Configuration menu - View commit details
-
Copy full SHA for 38be81c - Browse repository at this point
Copy the full SHA 38be81cView commit details -
Use handler specific memory estimate
In connection_context, we now use the handler-specific initial memory use estimate, rather than a single estimate for every handler type.
Configuration menu - View commit details
-
Copy full SHA for 5a9ca08 - Browse repository at this point
Copy the full SHA 5a9ca08View commit details -
Move session_resouces object to top level
The session_resources type was a private member of connection_context, but as we want to use it more broadly, move it out as a standalone public class. Additionally, pass it by shared_pointer in preparation for later changes will feed it into requests.
Configuration menu - View commit details
-
Copy full SHA for 958b933 - Browse repository at this point
Copy the full SHA 958b933View commit details -
Configuration menu - View commit details
-
Copy full SHA for 213b671 - Browse repository at this point
Copy the full SHA 213b671View commit details -
Introduce handler template for two-stage handlers
Single-stage handlers have a handler template which means that handler objects can be declared in a single line specifying their api object, min and max API versions. This change extends this nice concept to two-stage handlers as well.
Configuration menu - View commit details
-
Copy full SHA for d645c7a - Browse repository at this point
Copy the full SHA d645c7aView commit details -
Add the connection context to the memory estimator
Passing the connection context to the estimator allows the estimator to use the various subsystems to estimate the memory use of a given request.
Configuration menu - View commit details
-
Copy full SHA for bbea197 - Browse repository at this point
Copy the full SHA bbea197View commit details -
Use a better estimator for metadata requests
Currently we estimate that metadata requests take 8000 + rsize * 2 bytes of memory to process, where rsize is the size of the request. Since metadata requests are very small, this end up being roughly 8000 bytes. However, metadata requests which return information about every partition and replica may easily be several MBs in size. To fix this for metadata requests specifically, we use a new more conservative estimate which uses the current topic and partition configuration to give an upper bound on the size.
Configuration menu - View commit details
-
Copy full SHA for c218c2b - Browse repository at this point
Copy the full SHA c218c2bView commit details -
Include broker list in metadata estimation
Prior to this change, we used only the topic and partition data to estimate the size of the metadata response. Now, we also include the approximate size of the broker metadata portion of the response, which may be important if the list of brokers is very large or they have very long hostnames.
Configuration menu - View commit details
-
Copy full SHA for 6cb5a71 - Browse repository at this point
Copy the full SHA 6cb5a71View commit details -
Configuration menu - View commit details
-
Copy full SHA for f649f7f - Browse repository at this point
Copy the full SHA f649f7fView commit details -
Add NOLINT to use of operator[]
We already check the bounds so the cpp core guideline presumably does not apply and we don't want to pay the price for the additional bounds check inside at().
Configuration menu - View commit details
-
Copy full SHA for fd6353f - Browse repository at this point
Copy the full SHA fd6353fView commit details