mixin(Store): handle ResourceExhausted as a non-server error

[douglascamata]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

• Stopped treating ResourceExchausted as a server error. It's an error being used whenever a request is limited. Because of this, it shouldn't help trigger the ThanosStoreGrpcErrorRate alert.

Verification

[fpetkovski]

Thanks for the fix. Looks like the docs job failure is related to the changes.

[douglascamata]

@fpetkovski fixed in aa2ade6, thanks for the heads up.

[fpetkovski]

Do we want to have a panel for client errors as well? My only concern is that we have no good way to monitor this status code now.

[douglascamata]

@fpetkovski the dashboard is whole different problem that I plan to tackle soon. In this PR I only change alerts and rules for this reason.

We have a dashboard widget that shows different error timeseries (see below), but all labelled as "error". It's a pattern that repeats in grpc and http error charts. It's not that easy to fix it without breaking something else because there is a lot of reuse of functions (qpsErrTotalPanel, grpcErrorsPanel, and httpErrorsPanel) for this. My target is to make all these dashboards show a series for each error, properly labelled with their grpc_code or code (when applicable).

thanos/examples/dashboards/receive.json

Lines 148 to 152 in 2d6b0d4

	"expr": "sum by (job) (rate(http_requests_total{job=~\"$job\", handler=\"receive\",code=~\"5..\"}[$interval])) / sum by (job) (rate(http_requests_total{job=~\"$job\", handler=\"receive\"}[$interval]))",
	"format": "time_series",
	"intervalFactor": 2,
	"legendFormat": "error",
	"step": 10

[matej-g]

I agree it makes no sense to include this as server error, although I'm also wary of #6218 (review) (but sounds like @douglascamata has a plan 😎).

[douglascamata]

@matej-g @fpetkovski if I cannot find a good way of fixing all the widgets at once and for all, I will definitely patch out the gRPC errors widget to not account for ResourceExhausted and add a separate widget for it. Our team use the mixin dashboards, so we have lots of interest in keeping them in good shape. 👍

[matej-g]

Just some remaining nits 👍

[douglascamata]

@matej-g fixed all the suggestions myself in e9e0245 to use the docs/examples automatic generation to ensure nothing is forgotten. 👍

[douglascamata]

@matej-g FYI I opened #6231 to take care of make the error types more evident in gRPC (and HTTP) error widgets. 👍

[douglascamata]

@matej-g finally got the build to be all green, without any intermittent failure. PTAL. 🙇

[matej-g]

Small nit for changelog entry, otherwise good to go 🎸