Cancelled status code not reported in the gRPC server metrics

[ceremo]

Hi,

cancelled status codes are not reported in the gRPC server metrics, metrics in these cases are reported without the tag status code because GrpcServerObservationContext is not updated with the cancelled. For us it's important the gRPC server metric give us info of the status codes, and this could be improved for example modifying the onCancel method of the ObservationGrpcServerCallListener before running the stop operation, something like this:

  @Override
  public void onCancel() {
    try (final Scope scope = this.observation.openScope()) {
      super.onCancel();
    } finally {
      if (this.observation.getContext() instanceof final GrpcServerObservationContext obsCtx) {
        obsCtx.setStatusCode(Status.Code.CANCELLED);
      }
      this.observation.stop();
    }
  }

I have created a demo to reproduce easily cancelleds, the service has a delay of 5 seconds to give time to the client to cancel the request, you could make the call using postman or a simple grpc client with a deadline lower than 5 seconds.

Thanks for your time.

[ttddyy]

Observing the behavior of cancel, it seems the status code doesn't change even if the request is canceled.
Instead, the ServiceCall#isCancelled returns true for canceled requests.

So, I'm inclined to add GrpcServerObservationContext#isCancelled, which becomes a low cardinality tag, to indicate whether the request has been canceled or not, then keep the existing status code unchanged.

[ceremo]

It works for me, cancellations in webflux are defined using connectionAborted.

[JordiMartinezVicent]

Hi @marcingrzejszczak @ttddyy

With the soluction provided, it creates a new event with status cancelled but it could be useful to add the cancel information to the GrpcServerObservationContext which is used to retrieve the timer metric.

The same behavior is resolved in webflux adding a boolean at ServerRequestObservationContext

[jonatan-ivanov]

You mean you simply want this info added to the context even if it is not used here so that you can create an ObservationFilter or an ObservationHandler and use it?

Or do you want the instrumentation also add a low cardinality key-value about the fact of the cancellation (that can also mean removing the event)?

Either case, could you create a new issue and explain what you want and how you want to use this information?

I think right now you can hack this: it's not great but you can create an ObservationHandler that receives the cancellation event (onEvent) method and reacts to that (e.g.: sets a field/keyvalue on the context, etc).

[JordiMartinezVicent]

You mean you simply want this info added to the context even if it is not used here so that you can create an ObservationFilter or an ObservationHandler and use it?

Or do you want the instrumentation also add a low cardinality key-value about the fact of the cancellation (that can also mean removing the event)?

I think that both would be helpful. On the one hand, we want to differentiate that the timer metric generated has been cancelled. And on the other hand, we have a custom GrpcServerObservationConvention to add extra information, which would be helpful to know if the request has been cancelled.

And yes, at least in our case, the event would have no sense.

Either case, could you create a new issue and explain what you want and how you want to use this information?

Perfect!

I think right now you can hack this: it's not great but you can create an ObservationHandler that receives the cancellation event (onEvent) method and reacts to that (e.g.: sets a field/keyvalue on the context, etc).

I will explore it, thanks for the suggestion!

[JordiMartinezVicent]

@jonatan-ivanov I have just created the following issue #5301