Update github.com/thanos-io/objstore

[andyasp]

What this PR does

Updates objstore, which required code changes to bring forward. The motivation for the update is to bring in a GCS listing optimization.

A choice I made was to not expose the storage_connection_string option objstore added for Azure. The target use case for it seems to be testing with Azurite, so my gut feeling was that it wasn't currently necessary and it could always be added later.

Which issue(s) this PR fixes or relates to

Fixes: N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[andyasp]

Test failure looks legitimate and there's a data race in the logs too. I'll take a closer look tomorrow.

[pracucci]

A choice I made was to not expose the storage_connection_string option objstore added for Azure. The target use case for it seems to be testing with Azurite, so my gut feeling was that it wasn't currently necessary and it could always be added later.

LGTM. As you already said, we can always add it later if needed.

[andyasp]

Edit: I had an incorrect theory before and modified this post.

This looks like an issue with objstore (debatably mino-go) upstream.

Specifically, in putObjectMultipartStreamNoLength (this is being used since objstore introduced a io.nopCloser which can't be used to guess the size of the upload, which seems like its own issue) minio surprisingly modifies the UserMetadata map in the PutObjectOptions that is passed in:

	if !opts.SendContentMd5 {
		if opts.UserMetadata == nil {
			opts.UserMetadata = make(map[string]string, 1)
		}
		opts.UserMetadata["X-Amz-Checksum-Algorithm"] = "CRC32C"
	}

	// Initiate a new multipart upload.
	uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
	if err != nil {
		return UploadInfo{}, err
	}
	delete(opts.UserMetadata, "X-Amz-Checksum-Algorithm")

What map does objstore pass in? The putUserMetadata stored in the Bucket struct:

// Bucket implements the store.Bucket interface against s3-compatible APIs.
type Bucket struct {
	logger          log.Logger
	name            string
	client          *minio.Client
	defaultSSE      encrypt.ServerSide
	putUserMetadata map[string]string
	storageClass    string
	partSize        uint64
	listObjectsV1   bool
}

	if _, err := b.client.PutObject(
		ctx,
		b.name,
		name,
		r,
		size,
		minio.PutObjectOptions{
			PartSize:             partSize,
			ServerSideEncryption: sse,
			UserMetadata:         b.putUserMetadata,
			StorageClass:         b.storageClass,
			// 4 is what minio-go have as the default. To be certain we do micro benchmark before any changes we
			// ensure we pin this number to four.
			// TODO(bwplotka): Consider adjusting this number to GOMAXPROCS or to expose this in config if it becomes bottleneck.
			NumThreads: 4,
		},
	); err != nil {
		return errors.Wrap(err, "upload s3 object")
	}

[andyasp]

Opened thanos-io/objstore#77 and thanos-io/objstore#78

[andyasp]

The above PRs were merged and I revendored. The tests are now passing.

[56quarters]

LGTM, thank you!