Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for snappy compression in receivers #5575

Merged
merged 4 commits into from
Aug 8, 2022
Merged

Add support for snappy compression in receivers #5575

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3c954c8
Add support for snappy compression in receivers
fpetkovski Aug 6, 2022
2672762
Use snappy from klauspost/compress
fpetkovski Aug 8, 2022
5f30e37
Merge branch 'main' into snappy-compression
fpetkovski Aug 8, 2022
0226d44
Add copyright
fpetkovski Aug 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#5470](https://github.com/thanos-io/thanos/pull/5470) Receive: Implement exposing TSDB stats for all tenants
- [#5493](https://github.com/thanos-io/thanos/pull/5493) Compact: Added `--compact.blocks-fetch-concurrency` allowing to configure number of go routines for download blocks during compactions.
- [#5527](https://github.com/thanos-io/thanos/pull/5527) Receive: Add per request limits for remote write.
- [#5520](https://github.com/thanos-io/thanos/pull/5520) Receive: Meta-monitoring based active series limiting
- [#5520](https://github.com/thanos-io/thanos/pull/5520) Receive: Meta-monitoring based active series limiting.
- [#5575](https://github.com/thanos-io/thanos/pull/5575) Receive: Add support for gRPC compression.

### Changed

Expand Down
12 changes: 12 additions & 0 deletions cmd/thanos/receive.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"strings"
"time"

"google.golang.org/grpc"

extflag "github.com/efficientgo/tools/extkingpin"
"github.com/go-kit/log"
"github.com/go-kit/log/level"
Expand All @@ -29,6 +31,7 @@ import (
"github.com/thanos-io/objstore/client"
"gopkg.in/yaml.v2"

"github.com/thanos-io/thanos/internal/cortex/util/grpcencoding/snappy"
"github.com/thanos-io/thanos/pkg/block/metadata"
"github.com/thanos-io/thanos/pkg/component"
"github.com/thanos-io/thanos/pkg/exemplars"
Expand All @@ -48,6 +51,8 @@ import (
"github.com/thanos-io/thanos/pkg/tls"
)

const compressionNone = "none"

func registerReceive(app *extkingpin.App) {
cmd := app.Command(component.Receive.String(), "Accept Prometheus remote write API requests and write to local tsdb.")

Expand Down Expand Up @@ -139,6 +144,9 @@ func runReceive(
if err != nil {
return err
}
if conf.compression != compressionNone {
dialOpts = append(dialOpts, grpc.WithDefaultCallOptions(grpc.UseCompressor(conf.compression)))
}

var bkt objstore.Bucket
confContentYaml, err := conf.objStoreConfig.Content()
Expand Down Expand Up @@ -782,6 +790,7 @@ type receiveConfig struct {
replicaHeader string
replicationFactor uint64
forwardTimeout *model.Duration
compression string

tsdbMinBlockDuration *model.Duration
tsdbMaxBlockDuration *model.Duration
Expand Down Expand Up @@ -859,6 +868,9 @@ func (rc *receiveConfig) registerFlag(cmd extkingpin.FlagClause) {

cmd.Flag("receive.replica-header", "HTTP header specifying the replica number of a write request.").Default(receive.DefaultReplicaHeader).StringVar(&rc.replicaHeader)

compressionOptions := strings.Join([]string{compressionNone, snappy.Name}, ", ")
cmd.Flag("receive.grpc-compression", "Compression algorithm to use for gRPC requests to other receivers. Must be one of: "+compressionOptions).Default(compressionNone).EnumVar(&rc.compression, compressionNone, snappy.Name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should enable it by default here? 🙂

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree. Enabling it by default makes more sense.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to suggest that for the v1 release, but even now it would make sense given the favourable tradeoff. Updated to default 👍


cmd.Flag("receive.replication-factor", "How many times to replicate incoming write requests.").Default("1").Uint64Var(&rc.replicationFactor)

cmd.Flag("receive.tenant-limits.max-head-series", "The total number of active (head) series that a tenant is allowed to have within a Receive topology. For more details refer: https://thanos.io/tip/components/receive.md/#limiting").Hidden().Uint64Var(&rc.maxPerTenantLimit)
Expand Down
4 changes: 4 additions & 0 deletions docs/components/receive.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,10 @@ Flags:
--receive.default-tenant-id="default-tenant"
Default tenant ID to use when none is provided
via a header.
--receive.grpc-compression=none
Compression algorithm to use for gRPC requests
to other receivers. Must be one of: none,
snappy
--receive.hashrings=<content>
Alternative to 'receive.hashrings-file' flag
(lower priority). Content of file that contains
Expand Down