Skip other sections when reading metadata #826
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jtibshirani
commented
Sep 14, 2024
| @@ -126,11 +122,14 @@ func (r *reader) readTOC(toc *indexTOC) error { | |||
| return err | |||
| } | |||
|
|
|||
| skipSection := len(tags) > 0 && !slices.Contains(tags, tag) | |||
There was a problem hiding this comment.
Instead of introducing the "skip" concept, I could have taken advantage of the fact that the metadata sections are always first in the TOC. However, our index reading code is structured around flexible "section tags", and I got the feeling that section ordering wasn't an invariant we wanted to rely on.
jtibshirani
commented
Sep 14, 2024
| @@ -169,6 +174,27 @@ func (r *reader) readTOC(toc *indexTOC) error { | |||
| return nil | |||
| } | |||
|
|
|||
| func (r *reader) readHeader() (simpleSection, uint32, error) { | |||
There was a problem hiding this comment.
I factored out the first part of readTOC (now readTOCSections). This wasn't critical for the change.
jtibshirani
commented
Sep 14, 2024
| @@ -395,9 +421,9 @@ func (r *reader) readIndexData(toc *indexTOC) (*indexData, error) { | |||
| return &d, nil | |||
| } | |||
|
|
|||
| func (r *reader) readMetadata(toc *indexTOC) ([]*Repository, *IndexMetadata, error) { | |||
| func (r *reader) parseMetadata(metaData simpleSection, repoMetaData simpleSection) ([]*Repository, *IndexMetadata, error) { | |||
There was a problem hiding this comment.
Also simplified this method, as it's not a big deal to be copying simpleSection. Not critical for the change.
jtibshirani
force-pushed
the
jtibs/metadata
branch
from
2b05dff
to
508594e
Compare
jtibshirani
force-pushed
the
jtibs/metadata
branch
from
508594e
to
d055f00
Compare
Merged
jtibshirani
added a commit
that referenced
this pull request
Sep 19, 2024
Tiny follow up to #826. I resolved a conflict incorrectly and reverted a log line improvement.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Looking at heap profiles, the
ReadMetadatafunction creates a ton of garbage objects. The main contributor is in other sections from the TOC, specifically decodingcompoundSection.offsets. However, to read metadata, we only really need to parse the metadata sections.This PR introduces a
skipmethod that skips over a section without reading it. This greatly reduces the allocations fromReadMetadata: