Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added handling members with group.instance.id set i.e. static members #4684

Merged
merged 16 commits into from
May 20, 2022
Merged

Added handling members with group.instance.id set i.e. static members #4684

merged 16 commits into from
May 20, 2022

Commits on May 17, 2022

  1. k/group: added warn method to group context logger 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 17, 2022
    Configuration menu
    Copy the full SHA
    a334266 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2022

  1. k/generator: using named type for group.instance.id in request 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    477a2ba View commit details
    Browse the repository at this point in the history

  2. k/group: added generic checkpoint method 

    Checkpoint assignments may come from either the sync request or the
members metadata stored in group metadata. Added generic checkpoint
method that allows to build a checkpoint using different assignments
providers.

Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    27b2cb9 View commit details
    Browse the repository at this point in the history

  3. fixture: make stop_signal part of the fixture 

    Previously the `stop_signal` was destroyed in `fixture::start()` method
leading to the signaling stop immediately after fixture application was
started.

Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    a445bfd View commit details
    Browse the repository at this point in the history

  4. k/group: handle static membership protocol 

    Added handling members with `group.instance.id` set i.e. static members.

When static member joins the group its `member_id` is identified using a
mapping stored in `_static_members` map. This way broker can identify
the `member_id` that is assigned to requested instance id.
In dynamic membership protocol `member_id` is assigned by broker after
each restart of consumer (there is no state persistence on the client
side) hence every consumer restart causes rebalance since member with
new id joins the group. With static members, when `group.instance.id` is
present in the `JoinRequest` and mapping is already present coordinator
will use that mapping to find member metadata.

When using static membership protocol consumers does not send `LeaveGroup`
requests when stopping.

More information:
[KIP-345](https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances)

Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    10ac228 View commit details
    Browse the repository at this point in the history

  5. k/join_group: support static membership protocol in join group 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    578b1bb View commit details
    Browse the repository at this point in the history

  6. k/group_manager: support static membership in group manager 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    1a64134 View commit details
    Browse the repository at this point in the history

  7. k/offset_commit: support static membership in offset commit handler 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    b9295d9 View commit details
    Browse the repository at this point in the history

  8. k/describe_groups: support static membership in describe groups handler 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    abbfabd View commit details
    Browse the repository at this point in the history

  9. k/tests: added simple consumer group join test 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    745f999 View commit details
    Browse the repository at this point in the history

  10. tests: added kafka v3.0.0 cli tools support 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    c550f60 View commit details
    Browse the repository at this point in the history

  11. tests/rpk: added group instance id to group describe output 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    26f2c69 View commit details
    Browse the repository at this point in the history

  12. tests/kafka_cli_consumer: added formatter properties 

    Added `formatter_properties` parameter to control how messages are
formatted when using Kafka CLI tools consumer.

Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    ad3ac60 View commit details
    Browse the repository at this point in the history

  13. tests/kafka_cli_consumer: using graceful shutdown when stopping consumer 

    When consumer is being stopped it may send `LeaveGroup` request to the
broker. Using clean shutdown (SIGTERM) to give the consumer process
oportunity to finish gracefully.

Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    20d7ba5 View commit details
    Browse the repository at this point in the history

  14. tests/kafka_cli_consumer: added wait_for_started method 

    Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    e37d184 View commit details
    Browse the repository at this point in the history

  15. tests: added consumer group handling tests 

    Added tests validating basic consumer group operations when using
dynamic and static membership protocols.

Signed-off-by: Michal Maslanka <michal@vectorized.io>
    @mmaslankaprv
    mmaslankaprv committed May 20, 2022
    Configuration menu
    Copy the full SHA
    a91767e View commit details
    Browse the repository at this point in the history