ingest storage: per-query X-Read-Consistency HTTP header

[dimitarvdimitrov]

This PR builds on top of #7030

What this PR does

This PR allows to specify the consistency level of individual queries. This is controlled via the HTTP header X-Read-Consistency: strong|eventual. Because this is still experimental, it's not documented in user-visible docs (i.e. /docs)

X-Read-Consistency propagation

• Within a component the value is kept in the context and can be accessed via querierapi.ReadConsistencyFromContext
• Between Mimir components the header is propagated via an HTTP header or gRPC metadata

Within components

Propagating within components is somewhat trivial. It relies on the assumption that the original request context is retained and no place uses context.Background() (or an unrelated context).

Between components

Propagating between components is less trivial. Primarily because of creating phony requests, httpgrpc, and custom gRPC clients

• client --> query-frontend

  • the dskit HTTP server has a middleware which extracts the HTTP header and injects it into the HTTP request context

• query-frontend --> query-scheduler

  • query & query_range requests: the query-frontend adds the header to all httpgrpc requests
  • all other requests: the original HTTP request contains the HTTP header and that is injected in the httpgrpc request.
  • the query-scheduler treats httpgrpc requests as opaque, so it doesn't need to understand the header
  • Because of how all other requests are handler, the added metric cortex_query_frontend_queries_consistency_total{consistency=""} is only incremented for query & query_range requests, not for any other request

• query-scheduler/query-frontend --> querier

  • after receiving the httpgrpc request from the query-frontend the querier passes it to a prometheus HTTP router, same router uses the same middleware that the query-frontend uses. This middleware parses the HTTP header and injects it into the request context

• querier --> ingester

  • the ingester gRPC client has a unary interceptor which takes the consistency level from the context and injects it as gRPC metadata
  • the dskit gRPC server has a middleware which parses the metadata and injects the consistency level in the request context

[pracucci]

I'm mid review, but I need to leave now sorry. Just committing a comment.

[pracucci]

I think my comment is the opposite. Why not just always inject it? Also look at who else reads AuthMiddleware cause I think should inject it also when directly querying queriers for consistency.

[pracucci]

Why not just always inject it?

I mean, why only when auth is used? We could just always enable it. Then the fact many APIs currently don't support it is a separate discussion, but for consistency we should always enable it.

Alternatively, we could move to the other side of the specturum and just enable it for querier's API, so only for routes registered by RegisterQueryAPI().

I don't feel strong about this comment, but it still looks weird to bundle it to auth, given has nothing to do with auth.

[dimitarvdimitrov]

thanks for the review. Can you check out how dd76194 looks like?

[pracucci]

[nit] Have you considered calling it track....() and consistencyCounters -> track...() too?

[dimitarvdimitrov]

do you mean something like trackReadConsistency and trackRegexpMatchers? 👍 done in d98612e

[pracucci]

I just left a comment about a couple of commented lines of code. We don't have any e2e test (which I believe would be the only way to test everything is wiring correctly), but I don't think that's a blocker. We can move forward and we can add "adding e2e tests" to an issue for the future.