[v23.1.x] storage: make compaction_backlog more efficient for large segment counts
#11718
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It was unnecessary to accumulate all segments in the topic, since each term is calculated separately. Do the per-term calculation inline. This still has a memory footprint that scales linearly with the number of segments in a term, but not the total number of segments in a partition. (cherry picked from commit 73619eb)
This was an O(N^2) loop, which is gratuitously expensive for something that is only used to adjust I/O priorities. Clamp the number of segments that each segment will be compared to. This will have identical results in the typical case where the number of segments in a compacted topic term is low (because compaction itself keeps the number of segments low), while preventing O(N^2) runtime in the worst case of a user creating many segments in a single term. A realistic scenario would be for a user to write many thousands of segments to a topic while importing data, and then enable compaction at the end. Previously, this would have resulted in ~64M pow() calls for a partition with a terabyte of data in 128MiB segments. (cherry picked from commit 4f0197b)
The previous commit limited the computational complexity of the per-term calculation. This commit limits the max allocation size done while building the list of segments in a term. As with the previous commit, this makes no difference in typical compaction use cases, but protects us from pathological cases. (cherry picked from commit 730c8b4)
This was referenced Aug 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #11681
Fixes #11717,