Cortex Prometheus Frontend changed behaviour in 1.1.0

[itatabitovski]

With Cortex 1.0.0 it was possible to access the prometheus/thanos-query interface of -frontend.downstream-url by accessing http://<cortex>:9091/graph in your browser. This was convenient for quick queries and general access (Alerts,Rules,Configuration,Targets, Discovery, Stores, ...) to the backend.

In Cortex 1.1.0 this does not work any more and cortex returns 404 page not found.

[pstibrany]

Thanks for creating issue from this Slack discussion.

[pracucci]

@jtlisi This may be related to the API refactoring we did?

[pstibrany]

As discussed on that Slack thread, I don't think we ever wanted to proxy all paths to downstream Prometheus. So the question is: should we add that and make it a stable feature?

I'm fine with doing it under some specific path, e.g. /downstream/* would be forwarded to Prometheus (removing /downstream/ first). But then Prometheus needs to be configured with this option.

Having access to /graph may imply that Prometheus will use Cortex for queries -- but in fact, it will not, unless Cortex is configured with empty same /downstream/ API prefix.

I would prefer to support API forwards only (eg. for usage from Grafana) to avoid the extra complexity.

[midnightconman]

This breaks the prometheus link in grafana panel editor for people using cortex (I don't think that was the intended behavior)... is this a bug or a feature?

[pracucci]

This breaks the prometheus link in grafana panel editor for people using cortex (I don't think that was the intended behavior)... is this a bug or a feature?

Could you share a screenshot about which link is broken, please? I would like to better understand it.

[midnightconman]

Sure :D The prometheus link in this screenshot (on the right hand side):

This is grafana 6.6, but I think the link is the same in 7.x

[pracucci]

Sorry for the late reply.

We could map the /graph endpoint in the query-frontend, so that it would be passed to the downstream URL. This would solve the case of Grafana link, but not the general issue that any request is not proxied anymore.

@jtlisi This change was due to the single binary mode. The question is: should query-frontend (when running standalone) proxy any request to the downstream URL or not?

[bwplotka]

Ah thanks @yeya24 for linking. Yes. I think in Thanos version of query frontend we should be proxying rest to downstream, I think it's nice behavior of query frontend being transparent.

As per Cortex one, I believe once thanos query-frontend exists, Cortex can even disable downstream URL flag as it's no longer needed - thus reducing support burden of this path 🤔 Up to you (:

If still downstream URL needs to be supported, maybe we can share between Thanos & Cortex some passthrough handler as well as routing code (:

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.