Consumer offsets migration did not finish after an upgrade

[mmaslankaprv]

Problem description

During one of the cluster upgrade operations all consumers were not able to continue its operation. Consumers were 'stuck' and not being able to make progress.

Root cause of the problem was the fact that __consumer_offsets topic migration operation did not finish.

Consumer offsets migration is executed after all nodes in the cluster are updated and report that consumer_offsets feature is ready. For the time of migration we block all consumer groups operations. When migration is being executed all consumer group APIs will return NOT_COORDINATOR error. After all nodes finish migration all consumer groups state is available in new __consumer_offsets topic and all operations are being routed to it.

Problem analysis

One of the __consumer_offsets topic partition did not finish migrating preventing the whole migration step from finishing.
Following log entry was found in 2 out of 3 nodes:

INFO  2022-05-21 01:49:09,850 [shard 34] group-metadata-migration - group_metadata_migration.cc:377 - transforming data from {kafka_internal/group/24} to {kafka/__consumer_offsets/24} - is leader: false

This is expected behavior as on each node which host old kafka_internal/group topic replica a reconciliation loop is running checking if groups state recovered from migrated topic is exactly the same as the state of a groups when read from old kafka_internal/group topic.

Steps taken to investigate/fix the error:

• PR improving logging in consumer groups migration: Fix logging in group_metadata_migration #4902