You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During one of the cluster upgrade operations all consumers were not able to continue its operation. Consumers were 'stuck' and not being able to make progress.
Root cause of the problem was the fact that __consumer_offsets topic migration operation did not finish.
Consumer offsets migration is executed after all nodes in the cluster are updated and report that consumer_offsets feature is ready. For the time of migration we block all consumer groups operations. When migration is being executed all consumer group APIs will return NOT_COORDINATOR error. After all nodes finish migration all consumer groups state is available in new __consumer_offsets topic and all operations are being routed to it.
Problem analysis
One of the __consumer_offsets topic partition did not finish migrating preventing the whole migration step from finishing.
Following log entry was found in 2 out of 3 nodes:
INFO 2022-05-21 01:49:09,850 [shard 34] group-metadata-migration - group_metadata_migration.cc:377 - transforming data from {kafka_internal/group/24} to {kafka/__consumer_offsets/24} - is leader: false
This is expected behavior as on each node which host old kafka_internal/group topic replica a reconciliation loop is running checking if groups state recovered from migrated topic is exactly the same as the state of a groups when read from old kafka_internal/group topic.
Problem description
During one of the cluster upgrade operations all consumers were not able to continue its operation. Consumers were 'stuck' and not being able to make progress.
Root cause of the problem was the fact that
__consumer_offsets
topic migration operation did not finish.Consumer offsets migration is executed after all nodes in the cluster are updated and report that
consumer_offsets
feature is ready. For the time of migration we block all consumer groups operations. When migration is being executed all consumer group APIs will returnNOT_COORDINATOR
error. After all nodes finish migration all consumer groups state is available in new__consumer_offsets
topic and all operations are being routed to it.Problem analysis
One of the
__consumer_offsets
topic partition did not finish migrating preventing the whole migration step from finishing.Following log entry was found in 2 out of 3 nodes:
This is expected behavior as on each node which host old
kafka_internal/group
topic replica a reconciliation loop is running checking if groups state recovered from migrated topic is exactly the same as the state of a groups when read from oldkafka_internal/group
topic.Steps taken to investigate/fix the error:
The text was updated successfully, but these errors were encountered: