Add S3 part-size flag to configure the S3 minimum file size in bytes used for multipart uploads

[mlech-reef]

What this PR does

It adds S3 part-size flag to configure the S3 minimum file size in bytes used for multipart uploads

Which issue(s) this PR fixes or relates to

N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[dimitarvdimitrov]

thanks for the contribution! Can you share some details for how this config is helping? perhaps we can improve the default value and avoid increasing the config surface

[mlech-reef]

thanks for the contribution! Can you share some details for how this config is helping? perhaps we can improve the default value and avoid increasing the config surface

Thank you for quick replay. This parameter is already available in Thanos S3 provider, however, we would like to use Mimir.
I believe the default value is tweaked by Thanos for AWS S3, but if you like to use a different S3-compatible API, the other value can have more sense, depends on the provider, the network capacity and speed. I can not give exact value.
Another reason to have it, is that you can set it to minio maximum value 5GB to completely disable multipart uploads. This is actually our use case, as Backblaze B2 that we use doesn't support x-amz-checksum-* headers added by minio by default for multipart upload and this can not be disabled neither in Thanos and Mimir.

[dimitarvdimitrov]

LGTM. Thank you for working on this!