Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compact: add concurrency to meta sync #887

Merged
merged 5 commits into from
Mar 5, 2019

Conversation

mjd95
Copy link
Contributor

@mjd95 mjd95 commented Mar 5, 2019

Changes

Add a flag to the compact command to allow for fetching of the block metadata in parallel.

Verification

Tested by running thanos compact with --log.level=debug to see that it fetches the metadata much quicker when concurrency is added.

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Minor suggestions only.

@@ -92,6 +92,9 @@ func registerCompact(m map[string]setupFunc, app *kingpin.Application, name stri
maxCompactionLevel := cmd.Flag("debug.max-compaction-level", fmt.Sprintf("Maximum compaction level, default is %d: %s", compactions.maxLevel(), compactions.String())).
Hidden().Default(strconv.Itoa(compactions.maxLevel())).Int()

metaSyncConcurrency := cmd.Flag("meta-sync-concurrency", "Number of goroutines to use when syncing metadata from object storage.").
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be consistent with store gateway flag here as it is essentially the same thing.

Suggested change
metaSyncConcurrency := cmd.Flag("meta-sync-concurrency", "Number of goroutines to use when syncing metadata from object storage.").
metaSyncConcurrency := cmd.Flag("block-sync-concurrency", "Number of goroutines to use when syncing block metadata from object storage.").

@@ -92,6 +92,9 @@ func registerCompact(m map[string]setupFunc, app *kingpin.Application, name stri
maxCompactionLevel := cmd.Flag("debug.max-compaction-level", fmt.Sprintf("Maximum compaction level, default is %d: %s", compactions.maxLevel(), compactions.String())).
Hidden().Default(strconv.Itoa(compactions.maxLevel())).Int()

metaSyncConcurrency := cmd.Flag("meta-sync-concurrency", "Number of goroutines to use when syncing metadata from object storage.").
Default("1").Int()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can actually make it faster by default to some reasonable number like 20. (same as store gateway). There is no reason, NOT to.

@@ -108,6 +111,7 @@ func registerCompact(m map[string]setupFunc, app *kingpin.Application, name stri
name,
*disableDownsampling,
*maxCompactionLevel,
*metaSyncConcurrency,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's change var name as well to follow up flag name

pkg/compact/compact.go Show resolved Hide resolved
pkg/compact/compact.go Show resolved Hide resolved
pkg/compact/compact.go Show resolved Hide resolved
@bwplotka bwplotka merged commit 181c8ce into thanos-io:master Mar 5, 2019
@mjd95 mjd95 deleted the add-concurreny-to-meta-sync branch March 5, 2019 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants