Add parsing of v1 compaction index footers #8555
Conversation
An unfortunate side effect of introducing v2 compaction footers was that perfectly good indices with v1 footers would have to be rebuilt (after restart we have to verify index integrity to verify that the segment is already compacted and indices with v1 footers failed the version check). This commit adds compatibility code for v1 footer parsing.
| map(lambda p: len(p.segments) > count and p.recovered(), | ||
| partitions)) | ||
| class CompactionRecoveryUpgradeTest(RedpandaTest): | ||
| OLD_VERSION = (22, 2, 8) |
There was a problem hiding this comment.
I think latest redpanda_installer changes might have included the ability to just say (22, 2) and get the latest version from that series
There was a problem hiding this comment.
But then if we backport and release this changes in a 22.2 release, the test will stop working, right?
There was a problem hiding this comment.
That's true, but I don't think we will backport it to 22.2 -- format changes like this are for major releases.
| seg2mtime_1 = get_compacted_segment2mtime(to_restart) | ||
| assert len(seg2mtime_1) >= 3 | ||
|
|
||
| self.installer.install([to_restart], RedpandaInstaller.HEAD) |
There was a problem hiding this comment.
We should upgrade via the intervening versions: jumps straight from 22.2 up to HEAD might not work when head is some future release
There was a problem hiding this comment.
So what is the canonical way of doing this? I've browsed through #7310 , looks like it is still under discussion.
We could test just with upgrading specific versions but then it is kind of useless in dev.
There was a problem hiding this comment.
Yes, there is some missing infrastructure here:
- a helper for stepping through versions in an upgrade
- a definition in the test code of what the intervening versions are between a particular historical version and HEAD (like in https://github.com/redpanda-data/redpanda/pull/8253/files#diff-08ac9c7e3ca9bc4f0dcdb04ab5d5c90bd853ea2175df49ae895d538cff7a3268)
I don't think this has to be a blocker for this PR, but it will need revisiting later to make this test sustainable.
ztlpn
force-pushed
the
test-new-compaction-index-footer
branch
from
aafa588
to
9e387de
Compare
ztlpn
force-pushed
the
test-new-compaction-index-footer
branch
from
9e387de
to
467b931
Compare
|
/ci-repeat 1 |
|
Force-push: deflake the ducktape test. |
A followup to #7687
An unfortunate side effect of introducing v2 compaction footers was that perfectly good indices with v1 footers would have to be rebuilt and the segment re-compacted (after restart we have to verify index integrity to determine it the segment is already compacted and indices with v1 footers failed the version check). This PR adds compatibility code for v1 footer parsing.
Also added an end-to-end upgrade test that verifies that the new version doesn't re-compact segments processed by the old version, and the old version correctly re-compacts the segments compacted by the new version.
Backports Required
Release Notes