Google PubSub DD service name is hardcoded

[david-fennessey]

Environment:

• DD-Agent 1.35.0~97065ed022
• dd-trace-api: 1.35.0
• opentracing-api 0.33.0
• JVM 21
• Dockerfile + javaagent
• Spring Boot 3.2.5
• Spring Integration Core 6.3.0
• Spring Cloud GCP 5.4.1

Issue

When using the automated instrumentation for DataDog in JVM languages, when we use the Service Activator pattern within Spring Boot + PubSub, which is recommended by Google link, we notice that the spans do get correctly created, however, it looks like the DD Service name is hardcoded to always use google-pubsub as a service name link

This is not ideal because we would like the spring.consume operation to live inside the service which is calling that function, so that we can keep all of their operations (whether it be REST, GRPC, PubSub) inside one APM dashboard.

There is a workaround that we rolled out, which is to reassign the span upon receiving it, however, it'd be great if the pubsub decorator could respect the DD tags of the service.

import com.google.cloud.spring.pubsub.support.BasicAcknowledgeablePubsubMessage
import com.google.cloud.spring.pubsub.support.GcpPubSubHeaders
import datadog.trace.api.DDTags
import io.opentracing.Tracer
import org.springframework.integration.annotation.ServiceActivator

@Component
class MyConsumer(
    private val tracer: Tracer,
) {

  @ServiceActivator(inputChannel = PubSubChannelConfig.MY_CHANNEL)
  suspend fun consumeSearchExecutedEvent(
      payload: ByteArray,
      @Header(GcpPubSubHeaders.ORIGINAL_MESSAGE) message: BasicAcknowledgeablePubsubMessage
  ) {
    // Reassign the current span to our service
    val currentSpan = tracer.activeSpan()
    currentSpan.setTag(DDTags.SERVICE_NAME, "my-service-name")
    ...
   message.ack()
}
...

Our workaround makes it go from this

To this:

[davidgm0]

I'm facing the same issue. This is a problem because if you don't set the service name the traces don't show up in the dashboard of the service.

[amarziali]

Other instrumentations (like jms, kafka) have a so called "legacy tracing mode". By default, the receiver has the service name of the framework used (i.e. kafka or like in this case google-pubsub). When this is set to false, the real service name is used for the receiver span.

Google pubsub has no options to switch between those two modes. However there is a global flag to remove all those "synthetic" services and always use the same service name for all the span.

This is possible by adding -Ddd.trace.remove.integration-service-names.enabled=true to the jvm options.

Let us know if this will fit

[david-fennessey]

Thanks for the reply @amarziali! Speaking personally here, most of the integration boundaries automatically get detected perfectly, such as when I query elasticsearch, we see the request go into the elasticsearch APM integration which is super handy.

What I'm primarily asking for here is the second graph. Once I take the bytes out of pubsub, such as with my SearchExecutedConsumer.consumeSearchExecutedEvent I expect that the spans will show up under my service name's span, because that's the service that is doing the work.

The same thing happens in reverse on the flame graph for API calls to Elasticsearch, we see my service spans up to the point where Elasticsearch is called, and then it transfers the span ownership to elasticsearch, so that I can use DD to track all elasticsearch calls to all indices.

I'd prefer to not enable that integration-service-name feature, because that would also remove the outgoing API calls, such as the Elasticsearch integration detection.

In a way it works great for outgoing spans, but not inbound spans such as pubsub

[amarziali]

Thanks for the quick reply. I can keep track of this as feature request and see if I can quickly implement a legacy tracing style setting also for pubsub that will fit your needs

[david-fennessey]

Thank you, @amarziali!

[amarziali]

@david-fennessey I just opened a PR for it. The instruction to disable the legacy tracing are in the PR description. 🎁

[david-fennessey]

Thank you, @amarziali for the fast turnaround! I'll be happy to test it out when it's live :)

[amarziali]

No worries it will be available with 1.40.0. In the meantime if you wanna have a try you can also use the CI build (non prod ready)