receive/ruler: Segfault due to chunk bytes reuse.

[bwplotka]

	August 9th 2020, 18:01:27.296	fatal error: fault
	August 9th 2020, 18:01:27.296	unexpected fault address 0x7f594352d285
	August 9th 2020, 18:01:27.298		/home/bwplotka/.gvm/gos/go1.14.2/src/runtime/memmove_amd64.s:181 +0x147 fp=0xc0f83c78a8 sp=0xc0f83c78a0 pc=0x4688d7
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/types.pb.go:455 +0xc7 fp=0xc0f83c79c8 sp=0xc0f83c7960 pc=0xbf0297
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/rpc.pb.go:1336 +0xc3 fp=0xc0f83c7aa8 sp=0xc0f83c7a58 pc=0xbe3483
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).Marshal(0xc14d412430, 0x20ff300, 0xc14d412430, 0x7f57d538f1c0, 0xc14d412430, 0xc0f217d801)
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/rpc.pb.go:1315 +0x7a fp=0xc0f83c7b00 sp=0xc0f83c7aa8 pc=0xbe329a
	August 9th 2020, 18:01:27.298	google.golang.org/grpc/encoding/proto.codec.Marshal(0x20ff300, 0xc14d412430, 0x0, 0xc0f217d8f0, 0xc0f217d830, 0x3, 0x3)
	August 9th 2020, 18:01:27.298	[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f594352d285 pc=0x4688d7]
	August 9th 2020, 18:01:27.298	goroutine 231100606 [running]:
	August 9th 2020, 18:01:27.298		/home/bwplotka/.gvm/gos/go1.14.2/src/runtime/panic.go:1116 +0x72 fp=0xc0f83c7870 sp=0xc0f83c7840 pc=0x434b22
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/types.pb.go:419 +0x13e fp=0xc0f83c78f0 sp=0xc0f83c78a8 pc=0xbeffce
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/types.pb.go:565 +0x33f fp=0xc0f83c7960 sp=0xc0f83c78f0 pc=0xbf098f
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*Series).MarshalToSizedBuffer(0xc0e6882210, 0xc14d42e380, 0x34e, 0x34e, 0xbe57cd, 0xc0e6882210, 0x34b)
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalTo(0xc10e106ea8, 0xc14d42e380, 0x34e, 0x34e, 0x380, 0x1e15360, 0xc10e106e01)
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/rpc.pb.go:1346 +0x5c fp=0xc0f83c7a58 sp=0xc0f83c7a10 pc=0xbe353c
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).MarshalToSizedBuffer(0xc14d412430, 0xc14d42e380, 0x34e, 0x34e, 0x40a893, 0x1edf7c0, 0x20ff300)
	August 9th 2020, 18:01:27.298	
	August 9th 2020, 18:01:27.298	runtime.throw(0x21ad238, 0x5)
	August 9th 2020, 18:01:27.298	runtime.sigpanic()
	August 9th 2020, 18:01:27.298		/home/bwplotka/.gvm/gos/go1.14.2/src/runtime/signal_unix.go:702 +0x3cc fp=0xc0f83c78a0 sp=0xc0f83c7870 pc=0x44b5ec
	August 9th 2020, 18:01:27.298	runtime.memmove(0xc14d42e680, 0x7f594352d285, 0x19)
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*Chunk).MarshalToSizedBuffer(0xc14d40e1e0, 0xc14d42e380, 0x319, 0x34e, 0x173beec738b, 0x31c, 0x0)
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*AggrChunk).MarshalToSizedBuffer(0xc14d3e5780, 0xc14d42e380, 0x319, 0x34e, 0x33, 0x31a, 0x0)
	August 9th 2020, 18:01:27.298	github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalToSizedBuffer(0xc10e106ea8, 0xc14d42e380, 0x34e, 0x34e, 0xbe57cd, 0xc0e6882210, 0x34b)
	August 9th 2020, 18:01:27.298		/home/bwplotka/Repos/thanos/pkg/store/storepb/rpc.pb.go:1353 +0x79 fp=0xc0f83c7a10 sp=0xc0f83c79c8 pc=0xbe35f9
	August 9th 2020, 18:01:27.299	google.golang.org/grpc/encoding/proto.(*codec).Marshal(0x39de688, 0x20ff300, 0xc14d412430, 0x2660d00, 0xc0f21756e0, 0xc0f83c7c60, 0x8898d8, 0xc0f217d8f0)
	August 9th 2020, 18:01:27.299	google.golang.org/grpc.encode(0x7f57d9b0b6e0, 0x39de688, 0x20ff300, 0xc14d412430, 0x0, 0x1, 0xc0f217d8f0, 0xc0f83c7d38, 0x888fac)
	August 9th 2020, 18:01:27.299		/home/bwplotka/Repos/thanosgopath/pkg/mod/google.golang.org/grpc@v1.30.0/rpc_util.go:538 +0x52 fp=0xc0f83c7c28 sp=0xc0f83c7bc8 pc=0xa0dc42
	August 9th 2020, 18:01:27.299		/home/bwplotka/Repos/thanosgopath/pkg/mod/google.golang.org/grpc@v1.30.0/stream.go:1424 +0xfb fp=0xc0f83c7e48 sp=0xc0f83c7cc8 pc=0xa2235b
	August 9th 2020, 18:01:27.299		<autogenerated>:1 +0x50 fp=0xc0f83c7ec8 sp=0xc0f83c7e90 pc=0xa2c1e0
	August 9th 2020, 18:01:27.299		/home/bwplotka/Repos/thanos/pkg/store/multitsdb.go:234 +0x91 fp=0xc0f83c7f78 sp=0xc0f83c7f00 pc=0x19bc0c1
	August 9th 2020, 18:01:27.299	golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0d2289530, 

Triggered by

func (m *Chunk) MarshalToSizedBuffer(dAtA []byte) (int, error) {
	i := len(dAtA)
	_ = i
	var l int
	_ = l
	if len(m.Data) > 0 {
		i -= len(m.Data)
---->		copy(dAtA[i:], m.Data)
		i = encodeVarintTypes(dAtA, i, uint64(len(m.Data)))
		i--
		dAtA[i] = 0x12
	}
	if m.Type != 0 {
		i = encodeVarintTypes(dAtA, i, uint64(m.Type))
		i--
		dAtA[i] = 0x8
	}
	return len(dAtA) - i, nil
}

Root Cause: ChunkIterator gives chunk bytes which are... deallocated in the moment o marshalling. This happens especially when block was just created and head was compacted, thus deallocating byte range.

This might be best fixed on ChunkIter itself

[bwplotka]

prometheus/prometheus#7776

[bwplotka]

Fix on upstream: prometheus/prometheus#7776 (comment)