Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store-gateway: use bucket index instead of scanning the bucket #6808

Merged
merged 17 commits into from
Dec 4, 2023

Conversation

dimitarvdimitrov
Copy link
Contributor

This is a follow-up to #6673, similar change to #6779

@dimitarvdimitrov dimitarvdimitrov requested a review from a team as a code owner December 4, 2023 11:20
@dimitarvdimitrov dimitarvdimitrov marked this pull request as draft December 4, 2023 11:26
@dimitarvdimitrov dimitarvdimitrov marked this pull request as ready for review December 4, 2023 14:09
Copy link
Contributor

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, maybe one debug logging option left by mistake?

integration/store_gateway_limits_hit_test.go Outdated Show resolved Hide resolved
This is a follow-up to 6673, similar change to 6779

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
…eChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
…yInTheRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
@dimitarvdimitrov dimitarvdimitrov merged commit 1a459bb into main Dec 4, 2023
28 checks passed
@dimitarvdimitrov dimitarvdimitrov deleted the dimitar/st-gw/use-bucket-index branch December 4, 2023 14:49
@grafanabot
Copy link
Contributor

The backport to r266 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-6808-to-r266 origin/r266
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 1a459bb5efeec997ea21fc2a4442c6470dd20e9f
# Push it to GitHub
git push --set-upstream origin backport-6808-to-r266
git switch main
# Remove the local backport branch
git branch -D backport-6808-to-r266

Then, create a pull request where the base branch is r266 and the compare/head branch is backport-6808-to-r266.

grafanabot pushed a commit that referenced this pull request Dec 4, 2023
* store-gateway: use bucket index instead of scanning the bucket

This is a follow-up to 6673, similar change to 6779

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncWithDefaultShardingEnabled

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_SyncBlocks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSyncShouldRetryOnFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSync

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_Series_ShouldCorrectlyQuerySeriesSpanningMultipleChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_deleteLocalFilesForExcludedTenants

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SyncShouldKeepPreviousBlocksIfInstanceIsUnhealthyInTheRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SeriesQueryingShouldEnforceMaxChunksPerQueryLimit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix read-write mode integration tests

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Clean up users in integration tests more often

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Update CHANGELOG.md

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestGettingStartedWithGossipedRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix Test_MaxSeriesAndChunksPerQueryLimitHit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Remove debug option

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

---------

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
(cherry picked from commit 1a459bb)
@grafanabot
Copy link
Contributor

The backport to r266 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-6808-to-r266 origin/r266
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 1a459bb5efeec997ea21fc2a4442c6470dd20e9f
# Push it to GitHub
git push --set-upstream origin backport-6808-to-r266
git switch main
# Remove the local backport branch
git branch -D backport-6808-to-r266

Then, create a pull request where the base branch is r266 and the compare/head branch is backport-6808-to-r266.

dimitarvdimitrov added a commit that referenced this pull request Dec 4, 2023
* store-gateway: use bucket index instead of scanning the bucket

This is a follow-up to 6673, similar change to 6779

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncWithDefaultShardingEnabled

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_SyncBlocks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSyncShouldRetryOnFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSync

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_Series_ShouldCorrectlyQuerySeriesSpanningMultipleChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_deleteLocalFilesForExcludedTenants

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SyncShouldKeepPreviousBlocksIfInstanceIsUnhealthyInTheRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SeriesQueryingShouldEnforceMaxChunksPerQueryLimit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix read-write mode integration tests

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Clean up users in integration tests more often

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Update CHANGELOG.md

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestGettingStartedWithGossipedRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix Test_MaxSeriesAndChunksPerQueryLimitHit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Remove debug option

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

---------

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
(cherry picked from commit 1a459bb)
@grafanabot
Copy link
Contributor

The backport to r267 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-6808-to-r267 origin/r267
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 1a459bb5efeec997ea21fc2a4442c6470dd20e9f
# Push it to GitHub
git push --set-upstream origin backport-6808-to-r267
git switch main
# Remove the local backport branch
git branch -D backport-6808-to-r267

Then, create a pull request where the base branch is r267 and the compare/head branch is backport-6808-to-r267.

pr00se pushed a commit that referenced this pull request Dec 6, 2023
* store-gateway: use bucket index instead of scanning the bucket

This is a follow-up to 6673, similar change to 6779

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncWithDefaultShardingEnabled

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_SyncBlocks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSyncShouldRetryOnFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSync

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_Series_ShouldCorrectlyQuerySeriesSpanningMultipleChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_deleteLocalFilesForExcludedTenants

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SyncShouldKeepPreviousBlocksIfInstanceIsUnhealthyInTheRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SeriesQueryingShouldEnforceMaxChunksPerQueryLimit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix read-write mode integration tests

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Clean up users in integration tests more often

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Update CHANGELOG.md

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestGettingStartedWithGossipedRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix Test_MaxSeriesAndChunksPerQueryLimitHit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Remove debug option

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

---------

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
(cherry picked from commit 1a459bb)
pr00se added a commit that referenced this pull request Dec 6, 2023
#6839)

* store-gateway: use bucket index instead of scanning the bucket

This is a follow-up to 6673, similar change to 6779

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_InitialSyncWithDefaultShardingEnabled

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_SyncBlocks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSyncShouldRetryOnFailure

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_InitialSync

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_Series_ShouldCorrectlyQuerySeriesSpanningMultipleChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestBucketStores_deleteLocalFilesForExcludedTenants

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SyncShouldKeepPreviousBlocksIfInstanceIsUnhealthyInTheRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestStoreGateway_SeriesQueryingShouldEnforceMaxChunksPerQueryLimit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix read-write mode integration tests

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Clean up users in integration tests more often

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Update CHANGELOG.md

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix TestGettingStartedWithGossipedRing

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Fix Test_MaxSeriesAndChunksPerQueryLimitHit

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

* Remove debug option

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>

---------

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
(cherry picked from commit 1a459bb)

Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants