Receiver: reduce network bandwidth when ingesting data

[fpetkovski]

Is your proposal related to a problem?

We noticed that our intra-AZ egress costs tripled when we migrated from HA Prometheus pairs to receivers. We would have expected costs to go up by around 50% only due to the added replication factor.

Describe the solution you'd like

Looking at the code, it seems that forwarding requests between receivers is done by encoding remote-write requests in protobuf. This could be the potential reason and we might be able to reduce bandwith by encoding requests with snappy.

Describe alternatives you've considered

No alternatives found so far

[fpetkovski]

Cortex seems to support this already, so adding it to Thanos could be straightforward: cortexproject/cortex#2940

[bwplotka]

LGTM. thanks for noticing!

[squat]

cool. let's see. I didn't really understand how to read the network ingester graph from that issue as it has negative values for the bandwidth.

[fpetkovski]

Yeah the graphs are a bit strange. I'll give this thing a shot next week and measure changes in transmitted bytes from receivers.