query: implement active query tracker

[GiedriusS]

Is your proposal related to a problem?

If my Thanos Query is having latency issues while queries are running then I would like to be able to know what queries are being executed at that exact time.

Describe the solution you'd like

PromQL engine supports active query tracking by writing currently active queries to a file via mmap(2). I think we could add a new flag to enable support for this.

Describe alternatives you've considered

Using tracing and/or query logging. However, that data only appears after the fact. I would like to be able to know what queries are being executed when my Thanos Query/Store instances are being hammered.

Additional context

N/A

[matej-g]

This would be so awesome, internally within our team we already mentioned a couple of times it would be great to have feature like this. I had no idea PromQL engine supports this. @GiedriusS are you planning to work on this? I'm also wondering whether it would be possible to visualize this directly in the UI.

[metonymic-smokey]

@GiedriusS from a quick look into the promql lib, I think this feature will involve adding an additional engine option here.
The flag can be used to specify a dirname since an active query tracker requires that as shown here.

Please let me know if this is in the right direction. I could work on this once my current PR is merged, if no one else is working on it/plans to.
Thanks!

[GiedriusS]

Feel free to pick this up! It would be very cool to also show the currently active queries in the UI.

[yeya24]

Some other random ideas when I talked with Aditi about this yesterday. Those can be features to work on in the future.

1. Support Active Query Tracker on the Thanos Query Frontend rather than in each Thanos Querier.
2. Implement the Prometheus query tracker interface use something other than mmap to store the data
3. It is great to collect the whole query history, not only just the current active queries. Combined with point 2 this can be more powerful to visualize from the UI