Get rid of httpgrpc on distributor's write path

[duricanikolic]

What this PR does

This PR modifies the existing implementation of Distributor.Push() by returning only errors with gRPC codes coming from the distributor. The only exceptions are the errors returned by the ingester, which can still contain HTTP status codes. This will be fixed in one of the following PRs.

Proto messages WriteErrorDetails and ErrorCause field have been added to the mimirpb package.

enum ErrorCause {
  INVALID = 0;
  REPLICAS_DID_NOT_MATCH = 1;
  TOO_MANY_CLUSTERS = 2;
  BAD_DATA = 3;
  INGESTION_RATE_LIMITED = 4;
  REQUEST_RATE_LIMITED = 5;
}

message WriteErrorDetails {
  ErrorCause Cause = 1;
}

The purpose of WriteErrorDetails is to enrich distributor's gRPC errors with additional details, which closely explain the nature of the error itself. Possible values of ErrorCauses are analog to the distributor error types.

The following table shows the mapping between the error types encountered during Distributor.Push() and the corresponding gRPC codes:

error type	gRPC code	ErrorCause
validationError	codes.FailedPrecondition	mimirpb.VALIDATION
ingestionRateLimitedError	codes.ResourceExhausted	mimirpb.INGESTION_RATE_LIMITED
requestRateLimitedError	code.Unavailable or codes.ResourceExhausted	mimirpb.REQUEST_RATE_LIMITED
replicasDidNotMatchError	codes.AlreadyExists	mimirpb.REPLICAS_DID_NOT_MATCH
tooManyClustersError	codes.FailedPrecondition	mimirpb.TOO_MANY_CLUSTERS
other errors	codes.Internal	none

The table above does NOT include the errors returned by the ingester which contain either HTTP status codes 400 and 503 or gRPC codes: these errors are propagated in the original form. This will be improved in one of the following PRs.

Which issue(s) this PR fixes or relates to

Part of #6008

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[colega]

Opinionated nit, but this says the same, plus it's also checked by the compiler and can't get outdated.

Suggested change

	// requestRateLimitedError implements the distributorError interface.
	func (e requestRateLimitedError) errorCause() mimirpb.ErrorCause {
	var _ distributorError = requestRateLimitedError{}
	func (e requestRateLimitedError) errorCause() mimirpb.ErrorCause {

[duricanikolic]

I have implemented the required change.

[colega]

Can we add an INVALID = 0 const here? That would prevent the code from silently being interpreted as REPLICAS_DID_NOT_MATCH if there's a bug somewhere and cause isn't set.

[duricanikolic]

Sure, I can do it.

[replay]

What are these statements for? is that to verify that the type satisfies the interface?
It would be useful (for me) if there was a comment what this does.

[duricanikolic]

Hi @replay,
Exactly, these statements are here for verifying that the type satisfies the interface.
I was asked by @colega (here) to add them because this way the compiler complains if we by mistake forget to implement some methods of an interface.
I agree that adding a comment explaining that would be beneficial and will add it.

[colega]

I'm okay with having a comment, although this is a standard Golang statement.

[replay]

This looks good to me, thx