query-frontend: populate org_id in slow query logs

[neilfordyce]

The OrgId passed to Cortex is hard-coded to "fake":

thanos/cmd/thanos/query-frontend.go

Line 203 in 5aac477

).ServeHTTP(w, r.WithContext(user.InjectOrgID(r.Context(), "fake")))

When logs are produced to the slow query log, it results in logs which contain org_id=fake, e.g:
level=info ts=2020-09-16T13:33:14.670646Z caller=frontend.go:206 org_id=fake msg="slow query detected" method=GET host=localhost:10902 path=/api/v1/query_range time_taken=890.912705ms ...

When using Cortex Query Frontend (which populates org_id based on the contents of the X-Scope-OrgID header), there was extra value in these logs. The org_id field is useful for identifying sources of problematic queries which take a long time to execute.

Does anyone have any thoughts on populating that org_id log field with the contents of a request header?

[neilfordyce]

We have a few different clients which interact with Thanos, which provide various headers which are useful for finding the source of queries (e.g. X-Grafana-User which contains the user name of a logged in Grafana user). Would it be useful if Thanos query-frontend took a list of potential header keys to try to extract an OrgId, then fallback through them until it finds one which isn’t empty, something like query --query-frontend.org-id-headers=X-Grafana-User,X-Scope-OrgId,X-Another-Service-Header. If none of the headers are set then we could fallback to org_id=fake or anonymous.

It seems like it would be straightforward to add (and I'd be happy to contribute), but a lot depends how valuable other people find the slow query log.

[yeya24]

I hardcoded that value to fake because there is no org concept in Thanos. But I think supporting this is definitely a valid feature request. Any thoughts @kakkoyun @bwplotka?

[bwplotka]

Sounds good to me !

[neilfordyce]

Awesome. I'll have a look at getting a PR together for that.

[bwplotka]

I think this will work as short term solution, but we should think how to define multitenancy in Thanos long term. (: