Consider using sharded compactor from mimir-prometheus for vertical block splitting.

[fpetkovski]

Is your proposal related to a problem?

The Thanos compactor is currently not able to split blocks vertically during the compaction process. It will compact blocks until compact.block-max-index-size is reached, after which it will add a no-compaction marker and exclude that block from future compaction plans.

For high-churn use cases, we can often reach this limit very soon and end up with very small blocks that never get downsampled because they do not get reach the minimum 40h range.

For low-churn cases, we can end up compacting blocks that have a very long duration, which are then served by a single store-gateway instance instead of being distributed across many different ones.

Describe the solution you'd like

In a Slack discussion, @pstibrany suggested we look into the mimir-prometheus compactor which has a CompactWithSplitting method. We can use this component as a building block to do vertical-block splitting once the maximum index size of a block has been reached.

Describe alternatives you've considered

There is an old proposal to implement vertical block splitting: #4191. Whether we follow this approach or not, we can still use the mimir-prometheus compactor to avoid reimplementing block splitting.

Additional context

The mimir-prometheus codebase has an Apache2 license.
Link to Slack discussion: https://cloud-native.slack.com/archives/CL25937SP/p1668151046513269

Example of highly fragmented tenant blocks in S3 due to high churn leading to large index size for receiver shards: