store: concurrency query large series return vector cannot contain metrics with the same labelset

[mengskysama]

Thanos, Prometheus and Golang version used:

 ./thanos --version
thanos, version 0.18.0-rc.0 (branch: HEAD, revision: 0dde3ecf583fffec41fc6d2cdaef4f454095ddb5)
  build user:       root@50eca97f2102
  build date:       20210106-17:41:00
  go version:       go1.15.6
  platform:         linux/amd64

./prometheus --version
prometheus, version 2.24.0 (branch: HEAD, revision: 02e92236a8bad3503ff5eec3e04ac205a3b8e4fe)
  build user:       root@d9f90f0b1f76
  build date:       20210106-13:48:37
  go version:       go1.15.6
  platform:         linux/amd64

Object Storage Provider:

OSS

What happened:

query sum(increase(create_order{}[1d])) some time return {"status":"error","errorType":"execution","error":"vector cannot contain metrics with the same labelset"}

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

test1 func can reproduce

from gevent import monkey

monkey.patch_all(select=False)

import requests
import gevent

a = "/api/v1/query?query=" + 'sum(increase(create_order{}[1d]))'
url = "http://xxx:10902"


def req(r):
    ret = requests.get(r)
    print(ret.text)


def test1():
    # failed
    while True:
        gevent.joinall([
            gevent.spawn(req, url + a),
            gevent.spawn(req, url + a),
        ])


def test2():
    # ok
    while True:
        gevent.joinall([
            gevent.spawn(req, url + a),
        ])


test1()

output

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1610960171.214,"111"]}]}}

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1610960171.214,"111"]}]}}

{"status":"error","errorType":"execution","error":"vector cannot contain metrics with the same labelset"}

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1610960175.325,"111"]}]}}

Full logs to relevant components:

no error in store and query debug log

level=debug ts=2021-01-18T07:53:37.770001154Z caller=proxy.go:274 component=proxy request="min_time:1610870018000 max_time:1610956418000 matchers:<name:\"env\" xxx" >  aggregates:COUNTER " msg="Series: startedfanout streams" status="Store Addr: prod-thanos-playground002:10901 LabelSets: {replica=\"A\"} Mint: 1610862791323 Maxt: 9223372036854775807 queried;Store Addr: prod-thanos-store001:10901 LabelSets: {replica=\"A\"} Mint: 1610530654200 Maxt: 1610884414505 queried;Store Addr: prod-thanos-playground003:10901 LabelSets: {replica=\"A\"} Mint: 1610862789065 Maxt: 9223372036854775807 queried;Store Addr: prod-thanos-playground001:10901 LabelSets: {replica=\"A\"} Mint: 1610862789142 Maxt: 9223372036854775807 queried;Store Addr: prod-thanos-rule001:10901 LabelSets:  Mint: 1610769600000 Maxt: 9223372036854775807 queried"

Anything else we need to know:

• create_order ~ 7k series
• stop store or set --max-time<-1d work well

[GiedriusS]

Is it still reproducible after #3705?

[mengskysama]

Is it still reproducible after #3705?

Great
this pr can fix issue.

thanks!