feat swift: use ncs/swift and support large files for OpenStack Swift

Signed-off-by: Martin Chodur <m.chodur@seznam.cz>

CHANGELOG.md

@@ -24,6 +24,9 @@ We use *breaking* word for marking changes that are not backward compatible (rel
 - [#2667](https://github.com/thanos-io/thanos/pull/2667) Store: removed support to the legacy `index.cache.json`. The hidden flag `--store.disable-index-header` was removed.
 - [#2667](https://github.com/thanos-io/thanos/pull/2667) Compact: the deprecated flag `--index.generate-missing-cache-file` and the metric `thanos_compact_generated_index_total` were removed.
 - [2603](https://github.com/thanos-io/thanos/pull/2603) Store/Querier: Significantly optimize cases where StoreAPIs or blocks returns exact overlapping chunks (e.g Store GW and sidecar or brute force Store Gateway HA).
+- [#TBA](https://github.com/thanos-io/thanos/pull/TBA) Swift: Switched to a new library [ncw/swift]() providing large objects support.
+   By default, segments will be uploaded to the same container directory `segments/` if the file is bigger than `1GB`. 
+   To change the defaults see [the docs](./docs/storage.md#openstack-swift).
 
 ## [v0.13.0](https://github.com/thanos-io/thanos/releases) - IN PROGRESS
 

docs/storage.md

@@ -58,7 +58,7 @@ Current object storage client implementations:
 | [Google Cloud Storage](./storage.md#gcs) | Stable  (production usage)             | yes       | @bwplotka   |
 | [AWS/S3](./storage.md#s3) | Stable  (production usage)               | yes        | @bwplotka          |
 | [Azure Storage Account](./storage.md#azure) | Stable  (production usage) | no       | @vglafirov   |
-| [OpenStack Swift](./storage.md#openstack-swift)      | Beta  (working PoCs, testing usage)               | no        | @sudhi-vm   |
+| [OpenStack Swift](./storage.md#openstack-swift)      | Beta  (testing usage)               | no        | @FUSAKLA   |
 | [Tencent COS](./storage.md#tencent-cos)          | Beta  (testing usage)                   | no        | @jojohappy          |
 | [AliYun OSS](./storage.md#aliyun-oss)           | Beta  (testing usage)                   | no        | @shaulboozhiao,@wujinhu      |
 | [Local Filesystem](./storage.md#filesystem) | Beta  (testing usage)             | yes       | @bwplotka   |
@@ -297,16 +297,25 @@ config:
 
 ### OpenStack Swift
 
-Thanos uses [gophercloud](http://gophercloud.io/) client to upload Prometheus data into [OpenStack Swift](https://docs.openstack.org/swift/latest/).
+Thanos uses [ncw/swift](https://github.com/ncw/swift) client to upload Prometheus data into [OpenStack Swift](https://docs.openstack.org/swift/latest/).
 
 Below is an example configuration file for thanos to use OpenStack swift container as an object store.
 Note that if the `name` of a user, project or tenant is used one must also specify its domain by ID or name.
 Various examples for OpenStack authentication can be found in the [official documentation](https://developer.openstack.org/api-ref/identity/v3/index.html?expanded=password-authentication-with-scoped-authorization-detail#password-authentication-with-unscoped-authorization).
 
+By default, OpenStack Swift has a limit for maximum file size of 5 GiB. Thanos index files are often larger than that.
+To resolve this issue, Thanos uses [Static Large Objects (SLO)](https://docs.openstack.org/swift/latest/overview_large_objects.html)
+which are uploaded as segments. These are by default put into the `segments` directory of the same container.
+Default limit for using SLO is 1 GiB which is also maximal size of the segment.
+If you don't want to use the same container for the segments 
+(best practise is to use `<container_name>_segments` to avoid polluting listing of the container objects)
+you can use the `large_file_segments_container_name` option to override the default and put the segments to other container.
+
 [embedmd]:# (flags/config_bucket_swift.txt yaml)
 ```yaml
 type: SWIFT
 config:
+  auth_version: 0
   auth_url: ""
   username: ""
   user_domain_name: ""
@@ -321,6 +330,11 @@ config:
   project_domain_name: ""
   region_name: ""
   container_name: ""
+  large_object_chunk_size: 1073741824
+  large_object_segments_container_name: ""
+  retries: 3
+  connect_timeout: 10s
+  timeout: 5m
 ```
 
 ### Tencent COS

go.mod

@@ -23,7 +23,6 @@ require (
 	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
 	github.com/golang/snappy v0.0.1
 	github.com/googleapis/gax-go v2.0.2+incompatible
-	github.com/gophercloud/gophercloud v0.10.0
 	github.com/grpc-ecosystem/go-grpc-middleware v1.1.0
 	github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
 	github.com/hashicorp/golang-lru v0.5.4
@@ -34,6 +33,7 @@ require (
 	github.com/minio/minio-go/v6 v6.0.56
 	github.com/mozillazg/go-cos v0.13.0
 	github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
+	github.com/ncw/swift v1.0.52
 	github.com/oklog/run v1.1.0
 	github.com/oklog/ulid v1.3.1
 	github.com/olekukonko/tablewriter v0.0.2

go.sum

@@ -747,6 +747,8 @@ github.com/nats-io/nats.go v1.9.1/go.mod h1:ZjDU1L/7fJ09jvUSRVBR2e7+RnLiiIQyqyzE
 github.com/nats-io/nkeys v0.1.0/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxziKVo7w=
 github.com/nats-io/nkeys v0.1.3/go.mod h1:xpnFELMwJABBLVhffcfd1MZx6VsNRFpEugbxziKVo7w=
 github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
+github.com/ncw/swift v1.0.52 h1:ACF3JufDGgeKp/9mrDgQlEgS8kRYC4XKcuzj/8EJjQU=
+github.com/ncw/swift v1.0.52/go.mod h1:23YIA4yWVnGwv2dQlN4bB7egfYX6YLn0Yo/S6zZO/ZM=
 github.com/oklog/oklog v0.3.2/go.mod h1:FCV+B7mhrz4o+ueLpx+KqkyXRGMWOYEvfiXtdGtbWGs=
 github.com/oklog/run v1.0.0 h1:Ru7dDtJNOyC66gQ5dQmaCa0qIsAUFY3sFpK1Xk8igrw=
 github.com/oklog/run v1.0.0/go.mod h1:dlhp/R75TPv97u0XWUtDeV/lRKWPKSdTuV0TZvrmrQA=