Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add block compaction delay metric #7635

Merged
merged 8 commits into from
Apr 1, 2024

Conversation

jhalterman
Copy link
Member

@jhalterman jhalterman commented Mar 15, 2024

What this PR does

Adds a new summary metric to compactor, cortex_compactor_block_compaction_delay_seconds, which is the time between a block being created/updated and it being successfully compacted. The motivation for this metric is wanting to understand when there are delays in compacting blocks, such as from compactors being or restarting, that may indirectly cause non-compacted blocks to be queried via store-gateways.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@jhalterman jhalterman force-pushed the block-wait-time branch 4 times, most recently from 8b5ed9d to 254817a Compare March 16, 2024 01:09
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a first step, I'd prefer to avoid adding new field to struct representing meta.json.

pkg/compactor/bucket_compactor.go Outdated Show resolved Hide resolved
pkg/storage/tsdb/block/meta.go Outdated Show resolved Hide resolved
pkg/compactor/job.go Outdated Show resolved Hide resolved
@jhalterman jhalterman changed the title Add block compaction delay summary metric Add block compaction delay metric Mar 22, 2024
pkg/storage/tsdb/block/fetcher.go Outdated Show resolved Hide resolved
pkg/storage/tsdb/block/meta.go Outdated Show resolved Hide resolved
@jhalterman jhalterman force-pushed the block-wait-time branch 2 times, most recently from b40deef to 7e1efd8 Compare March 23, 2024 00:41
@jhalterman jhalterman marked this pull request as ready for review March 23, 2024 01:12
@jhalterman jhalterman requested a review from a team as a code owner March 23, 2024 01:12
@jhalterman jhalterman force-pushed the block-wait-time branch 6 times, most recently from 848ae2c to bfd792b Compare March 23, 2024 01:29
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Left some minor comments.

pkg/compactor/bucket_compactor.go Outdated Show resolved Hide resolved
pkg/compactor/bucket_compactor.go Outdated Show resolved Hide resolved
pkg/storage/tsdb/block/block.go Outdated Show resolved Hide resolved
pkg/storegateway/bucket.go Outdated Show resolved Hide resolved
pkg/compactor/bucket_compactor.go Outdated Show resolved Hide resolved
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. (We may want to mention new metric in the changelog too)

jhalterman and others added 8 commits April 1, 2024 11:17
- Change summary to a histogram
- Make LastUpdated field transient
- Fetch attributes only when compactor is fetching meta
- Use LastUpdated when filtering jobs
- remove LastUpdated from block meta
- change metric to minutes and use buckets with longer durations
- only fetch attributes for compaction job filtering and recording the new metric
Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
This reverts commit 55ef1ddd10b5b688a002b2cfb998055c92789b08.
@jhalterman jhalterman merged commit 96952cf into grafana:main Apr 1, 2024
29 checks passed
@jhalterman jhalterman deleted the block-wait-time branch April 1, 2024 23:04
@grafanabot
Copy link
Contributor

The backport to r282 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-7635-to-r282 origin/r282
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 96952cf7bbfbdbf5fa77f22032e834a12a8ac236
# Push it to GitHub
git push --set-upstream origin backport-7635-to-r282
git switch main
# Remove the local backport branch
git branch -D backport-7635-to-r282

Then, create a pull request where the base branch is r282 and the compare/head branch is backport-7635-to-r282.

jhalterman added a commit to jhalterman/mimir that referenced this pull request Apr 2, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)
jhalterman added a commit to jhalterman/mimir that referenced this pull request Apr 2, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)
@jhalterman jhalterman mentioned this pull request Apr 2, 2024
jhalterman added a commit that referenced this pull request Apr 2, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)
grafanabot pushed a commit that referenced this pull request Apr 3, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)
grafanabot pushed a commit that referenced this pull request Apr 3, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)
jhalterman added a commit that referenced this pull request Apr 3, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)

Co-authored-by: Jonathan Halterman <jonathan@grafana.com>
jhalterman added a commit that referenced this pull request Apr 3, 2024
Add block compaction delay summary metric

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
(cherry picked from commit 96952cf)

Co-authored-by: Jonathan Halterman <jonathan@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants