-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix duplicates consistency error by caching already translated offsets #5356
Commits on Jul 9, 2022
-
k/produce: do not use unknown_server_error
Kafka client doesn't process unknown_server_error correctly and it may lead to duplicates violating the idempotency. See the following issue for more info: https://issues.apache.org/jira/browse/KAFKA-14034 request_timed_out just like unknown_server_error means that the true outcome of the operation is unknown and unlike unknown_server_error it doesn't cause the problem so switching to using it to avoid the problem
Configuration menu - View commit details
-
Copy full SHA for 1a72446 - Browse repository at this point
Copy the full SHA 1a72446View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1dfd8d9 - Browse repository at this point
Copy the full SHA 1dfd8d9View commit details -
cluster: prepare partition for translating offset
Update all partition::replicate dependees which don't perform offset translation to bypass it via a direct raft reference
Configuration menu - View commit details
-
Copy full SHA for 67a3112 - Browse repository at this point
Copy the full SHA 67a3112View commit details -
Updating consumer groups to use conditional replication to prevent a situation when after a check a leadership jumps away, invalidates the check, jumps back just in time for the post check replication. check condition leadership goes to a new node the node replicates something which invalidates the conditions the leadership jumps back the node successfully replicates assuming that the condition is true Switched to a conditional replicate to fix the problem. When a group manager detects a leadership change it replays the group's records to reconstruct the groups state. We cache the current term in the state and use it as a condition on replicate. In this case we know that if the leadership bounce the replication won't pass.
Configuration menu - View commit details
-
Copy full SHA for e693bea - Browse repository at this point
Copy the full SHA e693beaView commit details -
c/types: introduce kafka offset types
We're going to mix raft and kafka offset in the same class, since both the offsets uses the same type it's easy to make an error and treat one as it was another. Introducing kafka offset to rely on the type system to prevent such errors.
Configuration menu - View commit details
-
Copy full SHA for 9335075 - Browse repository at this point
Copy the full SHA 9335075View commit details -
cluster: shift offset translation to partition
Shifting offset translation down the abstraction well to eventually reach rm_stm
Configuration menu - View commit details
-
Copy full SHA for e3d24d9 - Browse repository at this point
Copy the full SHA e3d24d9View commit details -
rm_stm: prepare to use kafka::offset based cache
Preparing rm_stm to use kafka::offset based seq-offset cache. Right now it uses raft offsets but there is a problem with it: once the cache items become older that the head of the log (eviction) panda becomes unable to use offset translation so we need to store already translated offsets. Since the cache is persisted as a part the snapshot so we need to change the disk format and provide backward compatibility. The change is splitted into two commits. Current commit introduces types to represent old format seq_cache_entry_v1 and tx_snapshot_v1 and adds compatibility machinary to convert old snapshot (tx_snapshot_v1) to new snapshot (tx_snapshot). The follow up commit updates the default types to use new format and updates the mapping between old and default types.
Configuration menu - View commit details
-
Copy full SHA for 63c5883 - Browse repository at this point
Copy the full SHA 63c5883View commit details
Commits on Jul 12, 2022
-
rm_stm: shift offset translation to rm_stm
switching to caching seq-kafka offsets cache to avoid out of range errors on translating offsets beyond the eviction point
Configuration menu - View commit details
-
Copy full SHA for 4b42c7e - Browse repository at this point
Copy the full SHA 4b42c7eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 065fb54 - Browse repository at this point
Copy the full SHA 065fb54View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9000762 - Browse repository at this point
Copy the full SHA 9000762View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e7346d - Browse repository at this point
Copy the full SHA 8e7346dView commit details -
Configuration menu - View commit details
-
Copy full SHA for e5846f1 - Browse repository at this point
Copy the full SHA e5846f1View commit details -
ducky: add fix 5355 upgrade test
manually validated the tests by tweaking active_snapshot_version() to ignore feature manager and to always use the newest version and checked that in this case the tests fail
Configuration menu - View commit details
-
Copy full SHA for c45672a - Browse repository at this point
Copy the full SHA c45672aView commit details