0.17.0 Query Frontend not presenting labels correctly

[isugimpy]

Going to make this simple. There's a fix already merged via #3444 but that didn't make the cut for the 0.17.0 release. Would it be possible to get a cherry picked bug fix release of 0.17.1 with this included? Without this, the payload just comes back as a bunch of nulls, which breaks label_values templating in Grafana.

[arvidsnet]

@metalmatze, would it be possible to release 0.17.1 with this fix included? Without the fix query-frontend-0.17 is not usable :(

[isugimpy]

@arvidsnet (and anyone else who sees this), the image tag master-2020-11-18-2d95f890 does include the fix if you don't mind using a build straight off of master, it'd just be nice to have this fixed up in a more visible and official way.

[yeya24]

Hi, thanks for reporting the issue. I cannot reproduce this issue so can you please provide more information?

1. What's your Grafana version? Do you see any error logs of it?
2. What's your configuration of query frontend?

[arvidsnet]

thank you, image with tag master-2020-11-18-2d95f890 solved it for me

[isugimpy]

@yeya24

1. Grafana 7.0.5, no errors.

        - query-frontend
        - --query-frontend.downstream-url=http://thanos-query.prometheus-future.svc.cluster.local:10902
        - --query-frontend.log-queries-longer-than=60s
        - |-
          --query-range.response-cache-config="config":
            "max_size": "512MB"
            "max_size_items": 0
            "validity": "6h"
          "type": "in-memory"

api/v1/series?match%5B%5D=kube_pod_info%7B%7D&start=1605728739&end=1605732339

Making the above API call on 0.17.0 (which is what the label_values template function in Grafana calls) to query-frontend returns a payload which looks like the following:

{"status":"success","data":[{"labels":null},{"labels":null},...]}

In my case, there are several hundred null label entries. Making the same API call directly to thanos-query returns a proper payload which looks like

{"status":"success","data":[{"__name__":"kube_pod_info","created_by_kind":"\u003cnone\u003e","created_by_name":"\u003cnone\u003e","endpoint":"http","host_ip":"10.x.x.x","instance":"10.x.x.x:8080","job":"kube-state-metrics","namespace":"kube-system","node":"ip-10-x-x-x.ec2.internal","pod":"etcd-ip-10-x-x-x.ec2.internal","pod_ip":"10.x.x.x","priority_class":"system-cluster-critical","prometheus":"prometheus-future/prometheus-future-promethe-prometheus","service":"prometheus-future-kube-state-metrics","uid":"f02a79d7-f685-11e9-bc8e-120bac4dcc64"},{"__name__":"kube_pod_info","created_by_kind":"\u003cnone\u003e","created_by_name":"\u003cnone\u003e","endpoint":"http","host_ip":"10.x.x.x","instance":"10.x.x.x:8080","job":"kube-state-metrics","namespace":"kube-system","node":"ip-10-x-x-x.ec2.internal","pod":"kube-apiserver-ip-10-x-x-x.ec2.internal","pod_ip":"10.x.x.x","priority_class":"system-cluster-critical","prometheus":"prometheus-future/prometheus-future-promethe-prometheus","service":"prometheus-future-kube-state-metrics","uid":"17095a86-79d4-11ea-a113-120bac4dcc64"},...]}

Making this same API call against the image I mentioned in my previous comment returns the same payload as thanos-query as expected.

[yeya24]

Thanks, I can reproduce this now with series API.

[wenerme]

Same issues here grafana/grafana#29271 , link for cross reference

[yeya24]

#3480 is already merged, the fix will be included in v0.17.1.

Let's wait it out and see if it solves this issue.

[yeya24]

Hi, this issue should be fixed in v0.17.1 and we have a new v0.17.2 now. I think this issue is addressed already so I ll close it.