-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DescribeGroups failed in ConsumerGroupTest.test_dead_group_recovery.static_members=True
#5079
Comments
Another instance: https://buildkite.com/redpanda/redpanda/builds/11148#01814a60-009d-4b49-9449-7ca334cb8379 (static_members=False) |
Also with static_members=False From PandaResults:
|
#5275 is a duplicate of this one |
The test is run on 3 nodes. The sequence is: create topic - produce and consume in a group - delete topic - restart all nodes - * - create topic. At the moment * After the restart of the nodes, there are two outcomes possible.
note over node1, node2, node3 : {kafka/__consumer_offsets/0} Recovered persistent state from kvstore: voted for: 2
...
test -> rpk : describe the-group --brokers 1,2,3
rpk -> node1 : FindCoordinator(the-group)
rpk <-- node1 : coordinator is node 2
rpk -> node1 : MetadataRequest
rpk <-- node1 : nodes: {1,2,3}, controller: 1, {kafka/__consumer_offsets/0} leader: 2
rpk -> node3 : MetadataRequest
rpk <-- node3 : nodes: {1,2,3}, controller: 1, {kafka/__consumer_offsets/0} leader: 2
test <-- rpk : coordinator: 2, state: dead
note over node1 : {kafka/__consumer_offsets/0} Recovered persistent state from kvstore: voted for: 1
note over node2 : {kafka/__consumer_offsets/0} Recovered persistent state from kvstore: voted for: 2 or 3
note over node3 : {kafka/__consumer_offsets/0} Recovered persistent state from kvstore: voted for: 3
...
node3 -> node2 : {kafka/__consumer_offsets/0} Vote request
node3 -> node1 : {kafka/__consumer_offsets/0} Vote request
node3 <-- node1 : vote_granted: 0 or 1
node3 <-- node2 : vote_granted: 1
note over node3 : becoming the leader
test -> rpk : describe the-group --brokers 1,2,3
rpk -> node2 : MetadataRequest
rpk <-- node2 : nodes: {1,2}, controller: 1, {kafka/__consumer_offsets/0} leader: 3
rpk -> node1 : FindCoordinator(the-group)
rpk <-- node1 : coordinator is node 3
rpk -> node2 : MetadataRequest
rpk <-- node2 : nodes: {1,2}, controller: 1, {kafka/__consumer_offsets/0} leader: 3
rpk -> node1 : MetadataRequest
rpk <-- node1 : nodes: {1,2,3}, controller: 1, {kafka/__consumer_offsets/0} leader: 3
test <-- rpk : "Kafka replied that group test-gr-1 has broker coordinator 3, but did not reply with that broker in the broker list" In each case, role of each of the nodes is the same. In the failure case, node2 does not list node3 because its |
Any thoughts on what the solution is? If it is an issue with test, redpanda, or kafka protocol in general? |
I can't see the root cause, but I've experienced issues with the correctness of |
@dlex is OOO for a week, fyi. |
If coordinator isn't available, we should retry, not throw. Fixes redpanda-data#5079
The rpk group_describe function is meant to be a retry loop, but wasn't correctly handling the case of the coordinator being unavailable -- I hit the same thing with scale testing, and had an un-merged fix handy. Opened #5610 with it. I haven't dissected the logs from this failure but I wouldn't be surprised if this became more likely when we recently tweaked the leader balancer to move more partitions at once. |
I think it is the same issue
|
@jcsp, ok let's see if that helps. Meanwhile I'm trying to run the older code with retry in the test itself to verify whether the coordinator reappears in the list in case of the failure. |
In my local tests, a single retry was enough to recover in all 3 instances where I have reproduced the error. |
I think I ended up the assignee here because of my previous change to the describe groups function, but am not actively working on it. |
When describing a group in a `test_dead_group` recovery test we must allow for RPK to return an error and then retry. Fixes: redpanda-data#5079 Signed-off-by: Michal Maslanka <michal@redpanda.com>
When describing a group in a `test_dead_group` recovery test we must allow for RPK to return an error and then retry. Fixes: redpanda-data#5079 Signed-off-by: Michal Maslanka <michal@redpanda.com>
Seen on v22.2.x https://buildkite.com/redpanda/redpanda/builds/18771#_ |
When describing a group in a `test_dead_group` recovery test we must allow for RPK to return an error and then retry. Fixes: redpanda-data#5079 Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 47711c8)
https://buildkite.com/redpanda/redpanda/builds/11093#018144a0-4373-4133-b2d2-09624ddc39bd
The text was updated successfully, but these errors were encountered: