Default partitions count for __consumer_offsets topic increased
#5412
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a consumer tries to locate a consumer group coordinator of a cluster for the first time, the __consumer_offsets topic is created with the number of partitions as per the group_topic_partitions property. The default value for that property was 1 which means that unless a different value was explicitly specified by the customer at a very early stage of cluster's life, all OffsetCommit requests from all consumers will be going to a single broker. This change increases the default value to 16 as a reasonable trade-off between OffsetCommit parallelism for the clusters that will use consumer groups later in their life, and the overhead for the clusters that won't use consumer groups. redpanda-data#5222
|
@dlex are there any breaking changes from a user perspective here. |
|
this lgtm, although it seems that some of the tests are based on assumption that there is only one partition in |
|
@emaxerrno no user breaking changes. This change only affects new cluster and (rare?) cases of existing clusters where consumer groups have not been used yet. |
test_leadership_transfer relies on kafka_group_offset metric source as an indication of the node being a group coordinator. However in admin API there is no way to know which __consumer_groups partition a specific consumer group uses, and the testcase always works with partition 0. To make that work, number of __consumer_groups partition is set to 1 explicitly.
dlex
force-pushed
the
5222_consumer-offsets-topic-default-partitioning
branch
from
16128a4
to
20c701a
Compare
dotnwat
approved these changes
Jul 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cover letter
When a consumer tries to locate a consumer group coordinator of a cluster for the first time, the
__consumer_offsetstopic is created with the number of partitions as per thegroup_topic_partitionsproperty. The default value for that property was 1 which means that unless a different value was explicitly specified by the customer at a very early stage of cluster's life, all OffsetCommit requests from all consumers will be going to a single broker. This change increases the default value to 16 as a reasonable trade-off between OffsetCommit parallelism for the clusters that will use consumer groups later in their life, and the overhead for the clusters that won't use consumer groups.Fixes #5222
Release notes
Improvements