net: add conn_quota for connection count limits
#3901
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jcsp
force-pushed
the
connection-count-limits
branch
5 times, most recently
from
489b0f3
to
32ed576
Compare
jcsp
requested review from
dotnwat,
NyaliaLui,
BenPope and
mmaslankaprv
as code owners
jcsp
force-pushed
the
connection-count-limits
branch
from
32ed576
to
a984b8c
Compare
dotnwat
reviewed
Mar 9, 2022
jcsp
force-pushed
the
connection-count-limits
branch
from
a984b8c
to
67cc890
Compare
|
CI failure was known issue #3924 |
dotnwat
reviewed
Mar 12, 2022
jcsp
force-pushed
the
connection-count-limits
branch
from
67cc890
to
456efba
Compare
|
This had a rather unlucky arm64 run with three (!) unrelated test failures:
|
dotnwat
reviewed
Mar 14, 2022
src/v/net/conn_quota.cc
Outdated
| }) | ||
| .finally([u = std::move(units)] {}) | ||
| // Keep allowance alive until after units are dropped | ||
| .finally([allowance, u = std::move(units)] {}); |
There was a problem hiding this comment.
did you intend to remove the units capture in the second finally block? it should be a noop, but clangd should also warn here about use-after-move.
There was a problem hiding this comment.
Yes, this was a mistake. Fixed.
This helper was previously only used in a test that didn't set a non-empty default value, but it shouldn't have been ignoring the value passed to constructor.
These settings will enable limiting the number of concurrent connections per broker, or per client IP per broker.
This is just a pass through to the equivalent semaphore method. It's useful if calling from a context where you know you won't sleep.
This is the logic for tracking node-wide allowances for how many connections may be opened, or how many connections may be opened per client IP.
This is the first unit test definition in src/v/net, so comes with a new CMakeLists etc.
This counts how many connections we dropped for hitting a connection count limit.
I'm about to use it in connection_limits_test.
This is a ducktape test for exercising the new configurable connection count limits.
This fixes the potential crash from clear() getting called in the background while reclaim_to or cancel_reclaim_to was holding the mutex in home_allowance. remote_allowance doesn't need this special treatment.
jcsp
force-pushed
the
connection-count-limits
branch
from
456efba
to
cab9d6f
Compare
dotnwat
approved these changes
Mar 14, 2022
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
RFC: #3744
The main complexity here is the cross-shard coordination of limits that apply at a per-node level. Each client IP has a "home" shard where their
home_allowancelives, and other shards can "borrow" some tokens in theirremote_allowance. To avoid starvation when many tokens have been borrowed, there is a "reclaim" mechanism whereby the home shard reclaims all borrowed tokens, and sets a flag on the remote shards to dispatch any freed tokens back to the home shard.In normal operation where the limit is comfortably high enough for the workload, all shards that are handling connections will end up with a few tokens in their borrowed pool, and service incoming requests without having to do cross-shard communication. Cross-shard communication only kicks in when there is some pressure for the tokens.
This PR does not add the "overrides" variant of the config for setting limits on individual addresses, that is future work.
These settings are similar to Kafka's
max.connections.per.ipandmax.connectionssettings.Connection limits are only applied to the Kafka protocol server, not the internal RPC server.
Fixes #3698
Release notes
Features
kafka_connections_maxandkafka_connections_max_per_ipare added to control this behavior. By default both properties are switched off (i.e. the connection count is unlimited). These limits apply on a per-node basis. Note that the number of connections is not exactly equal to the number of clients: clients typically open 2-3 connections each.