[v22.3.x] Backport of #5315 #7603 #7923 #7604 #7814 #8016 #8019
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
On stderr it gets mixed with logs. On stdout it is straightforward to capture and decode json for use in tests. (cherry picked from commit e168164)
(cherry picked from commit 9f00c02)
This was wasteful: all messages from before the snapshot should already be accounted for in the state we loaded from the snapshot. (cherry picked from commit 963fc3d)
A None value is a deletion. Previously, we ignored these and thereby the kvstore dump would include all keys ever created, even if they had since been deleted. (cherry picked from commit 7f666f7)
(cherry picked from commit b492385)
…ing functions (cherry picked from commit 37b76a1)
…batch.header.type instead of batch.type) (cherry picked from commit 18328b1)
... and augmented Reader.read_envelope to accept a type_read Callable[rdr: Reader, version: Int]: Dict and a max_version parameter.
If a type_read is used, it is invoked with reader and envelope if envelope.version <= max_version parameter.
the result is joined with a dict {'envelope': envelope} and returned, otherwise an error dict is returned.
If no type_read is used, envelope is returned immediately
(cherry picked from commit 4bbd0c5)
the new code tries to mimic the structure of the adl decoding functions. the new code is aware of the version number embedded in the binary stream and will skip the field if the version is not supported, while reporting the error. (cherry picked from commit 021317a)
(cherry picked from commit ce43d16)
(cherry picked from commit 8b924de)
…luster_config_command (cherry picked from commit 29f9e7e)
…ent_command (cherry picked from commit 98e4df6)
…ytes (cherry picked from commit 8fe0654)
…are processed
in case of unread byte, an 'unread': {'k': ..., 'v': ...} field is added and an error is logged
this check is not exact but is a good proxy to know if the decoding code is correct
(cherry picked from commit ba26cd7)
(cherry picked from commit d8d85c4)
(cherry picked from commit a9ab9fa)
(cherry picked from commit 8557ac4)
(cherry picked from commit a7e6983)
Partitions have various kvstore items, stored by the raft + storage layers. When the topic is deleted, they should be deleted too. Use our offline log decoder tool to inspect the kvstore out-of-band and validate the keys go away when the topic is deleted. (cherry picked from commit fac7d51)
(cherry picked from commit 1813135)
This needs fixing, but better to comment it out and have the rest of the command work. (cherry picked from commit 189e475)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 065d1f4)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 1f6da22)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 8db3de7)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit d7d49aa)
Redpanda maintains on disk format compatibility with Kafka for `__consumer_offsets` topic messages. Added support for `expiry_timestamp` field that was part of `OffsetCommitValue` version 1. Previously redpanda implementation defaulted to the most recent version 3. Expiry timestamp may be set by clients using older versions of `OffsetCommit` requests. To maintain compatibility we must persist the value for older messages. Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 913a5f7)
Introduced tool that may be helpful when dealing with consumer offsets topic issue. The tool is designed to make changing the `__consumer_offsets` topic partition count. It can be also used to restore consumer group state. Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 82d136a)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 28c5bae)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 34e3ff2)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 2f476fa)
Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 428a488)
mmaslankaprv
changed the title
|
curiously test_cloud_storage_rpunit failed on both release and debug. |
|
Looks like known CI failure: |
dotnwat
previously approved these changes
Jan 5, 2023
I thought that we are going to backport some other parts that require this feature, if not i will drop it |
maybe, but i was thinking that reason we weren't shipping offset retention on Jan 6 was because we were going to wait until 23.1. maybe a good thing for slack discussion? |
This reverts commit 6421704. This is not being backported.
|
Failure is k8s and #7973 |
dotnwat
approved these changes
Jan 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #5315
Backport of PR #7603
Backport of PR #7604
Backport of PR #7923
Backport of PR #7814
Backport of PR #8016