Introduce BucketProvider interface #596
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hiddeco
added
enhancement
hiddeco
force-pushed
the
bucket-provider-interface-dev
branch
2 times, most recently
from
cac61ff
to
3a31d1c
Compare
This was referenced Feb 28, 2022
hiddeco
force-pushed
the
bucket-provider-interface-dev
branch
from
3a31d1c
to
380dd32
Compare
darkowlzz
approved these changes
Feb 28, 2022
pkg/minio/minio_test.go
Outdated
| @@ -0,0 +1,284 @@ | |||
| /* | |||
| Copyright 2021 The Flux authors | |||
There was a problem hiding this comment.
New file:
Suggested change
| Copyright 2021 The Flux authors | |
| Copyright 2022 The Flux authors |
darkowlzz
reviewed
Feb 28, 2022
hiddeco
force-pushed
the
bucket-provider-interface-dev
branch
from
380dd32
to
9fda293
Compare
This commit introduces a BucketProvider interface for fetch operations against object storage provider buckets. Allowing for easier introduction of new provider implementations. The algorithm for conditionally downloading object files is the same, whether you are using GCP storage or an S3/Minio-compatible bucket. The only thing that differs is how the respective clients handle enumerating through the objects in the bucket; by implementing just that in each provider, I can have the select-and-fetch code in once place. The client implementations do now include safe-guards to ensure the fetched object is the same as metadata has been collected for. In addition, minor changes have been made to the object fetch operation to take into account that: - Etags can change between composition of index and actual fetch, in which case the etag is now updated. - Objects can disappear between composition of index and actual fetch, in which case the item is removed from the index. Lastly, the requirement for authentication has been removed (and not referring to a Secret at all is thus allowed), to provide support for e.g. public buckets. Co-authored-by: Hidde Beydals <hello@hidde.co> Co-authored by: Michael Bridgen <michael@weave.works> Signed-off-by: pa250194 <pa250194@ncr.com>
hiddeco
force-pushed
the
bucket-provider-interface-dev
branch
from
9ed66bb
to
ed6c6eb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a BucketProvider interface for fetch operations
against object storage provider buckets. Allowing for easier
introduction of new provider implementations.
The algorithm for conditionally downloading object files is the same,
whether you are using GCP storage or an S3/Minio-compatible
bucket. The only thing that differs is how the respective clients
handle enumerating through the objects in the bucket; by implementing
just that in each provider, I can have the select-and-fetch code in
once place.
The client implementations do now include safe-guards to ensure the
fetched object is the same as metadata has been collected for. In
addition, minor changes have been made to the object fetch operation
to take into account that:
which case the etag is now updated.
in which case the item is removed from the index.
Lastly, the requirement for authentication has been removed (and not
referring to a Secret at all is thus allowed), to provide support
for e.g. public buckets.
Supersedes #455