Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initsync skip local blobs #13827

Merged
merged 10 commits into from
Apr 5, 2024
Merged

Initsync skip local blobs #13827

merged 10 commits into from
Apr 5, 2024

Conversation

kasey
Copy link
Contributor

@kasey kasey commented Mar 28, 2024

What type of PR is this?
Feature

What does this PR do? Why is it needed?

Modify blob request methods to skip slot ranges where we already have blobs on disk. Also modifies the availability store so that if a blob is visible in the pruner cache, it does not require that index to be in the availability store cache, only verifying and savings blobs that are not yet known to the cache.

Base automatically changed from expose-blob-cache to develop April 1, 2024 14:26
@kasey kasey force-pushed the initsync-skip-local-blobs branch from 9fa1151 to 2e7b7e2 Compare April 1, 2024 19:55
@kasey kasey marked this pull request as ready for review April 3, 2024 14:26
@kasey kasey requested a review from a team as a code owner April 3, 2024 14:26
@kasey kasey force-pushed the initsync-skip-local-blobs branch from f76e31c to 5afcca0 Compare April 3, 2024 14:32
@kasey kasey enabled auto-merge April 5, 2024 17:21
Comment on lines 98 to 101
if err == nil {
entry.setDiskSummary(sumz.Summary(root))
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do something about err != nil

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The summarizer is an optional optimization. If for whatever reason the summarizer didn't successfully init, we just don't use it. But a debug log would make sense.

Comment on lines 429 to 436
for z := len(fc) - 1; z > i; z-- {
hcz := fc[z]
if bs.Summary(hcz.root).AllAvailable(hcz.count) {
continue
}
needed.high = hcz.slot
return needed
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this assume that fc is a sorted list? If so, is that a safe assumption?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kasey kasey added this pull request to the merge queue Apr 5, 2024
Merged via the queue into develop with commit 04f231a Apr 5, 2024
17 checks passed
@kasey kasey deleted the initsync-skip-local-blobs branch April 5, 2024 19:16
@kasey kasey mentioned this pull request Apr 11, 2024
nisdas pushed a commit that referenced this pull request Jul 4, 2024
* wip - init-sync skip available blob req

* satisfy deep source

* gaz

* don't need to sort blobs; simplify blobRequest stack

* wip debug log to watch blob skip behavior

* unit tests for new blob req generator

* refactor to reduce blob req func count

* log when WaitForSummarizer fails

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants