[exporter/datadog] Update default logic for service entry spans identification

[liustanley]

What will change?

Datadog has a notion of service entry spans, which allows users to filter on a subset of traces and indicates that trace metrics should be generated. We are introducing the following mapping in order to consistently identify service entry spans from OTLP spans from the Datadog Exporter.

OpenTelemetry Convention	Datadog Convention
Root span	Service entry span
Server span (span.kind: server)	Service entry span
Consumer span (span.kind: consumer)	Service entry span
Client span (span.kind: client)	Generate trace metrics
Producer span (span.kind: producer)	Generate trace metrics
Internal span (span.kind: internal)	No trace metrics generated

Who is affected?

Most users will see an increase in trace metrics with this change, which may affect existing monitors that are based on trace metrics. Users who only have internal spans will see a decrease in trace metrics.

If you are using compute_stats_by_span_kind, you likely won't be affected because we are also computing stats by span kind in this change.

What should I do if I am affected?

If you have existing monitors based on trace metrics, you can update them after upgrading since this change will introduce more consistency in trace metrics. If you only have internal spans, please update your instrumentation according to the above table to receive trace metrics and service entry spans. The transform processor can also be used to update span kind if needed.

When will it change?

This change will first be released as an opt-in feature in order to avoid breaking changes for customers. This feature has already been merged in the datadog-agent repository and is pending release (PR). Once version 7.53.0 of the agent is released (tentatively mid April), this change will be implemented in collector-contrib. Once this feature has been deemed stable, we will enable this change by default.

[dudo]

Internal span (span.kind: internal) No trace metrics generated

Is there a mechanism in place to create trace metrics for these, ad hoc?

[liustanley]

Internal span (span.kind: internal) No trace metrics generated

Is there a mechanism in place to create trace metrics for these, ad hoc?

Not at the moment, but this is something we can consider if there are valid use cases for this. We only intend on computing trace metrics for incoming and outgoing spans, which is why we recommend updating your instrumentation according to the table above to receive trace metrics and service entry spans.

[dudo]

With the datadog agent (I think that's where it happens, anyway) you can pass _dd.measured as a tag on the span to trigger trace metrics. Could we do something similar with OTel, or do you think it'd be abused?

I guess we could re-type the spans, too. A concrete case for this is with something like graphql.internal. It's not a server span, per se, since the webserver has to receive the request first, but it's still the logical edge of the system. Is that reasonable?

edit: Another case that's more ubiquitous is something like cron.internal. This is the edge of the system, technically, and something that really should be the root span of a trace.

[liustanley]

With the datadog agent (I think that's where it happens, anyway) you can pass _dd.measured as a tag on the span to trigger trace metrics. Could we do something similar with OTel, or do you think it'd be abused?

I guess we could re-type the spans, too. A concrete case for this is with something like graphql.internal. It's not a server span, per se, since the webserver has to receive the request first, but it's still the logical edge of the system. Is that reasonable?

edit: Another case that's more ubiquitous is something like cron.internal. This is the edge of the system, technically, and something that really should be the root span of a trace.

Are graphql.internal and cron.internal typically root spans of their traces? If so, they will still get identified as service entry spans. We would prefer re-typing spans over using _dd.measured as an override, but this is something we can consider as well.

Also as an FYI, we changed this feature to be opt-in to avoid breaking changes for customers, and we will enable it by default in the future.

[dudo]

In those 2 examples, graphql would be behind a webserver, so not typically the root span. Cron would be, so good call on that.

I guess for spans we really want to keep, we just re-type. That's a straight forward enough solution. I know this wasn't directly related to your post, but thanks for answering my questions!

[github-actions]

Pinging code owners for exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Removing needs triage as this was filed by a code owner.

[liustanley]

This change has been released as an opt-in feature as a part of opentelemetry-collector-contrib v0.100.0 (PR). Please refer to the following documentation for usage instructions: https://docs.datadoghq.com/opentelemetry/schema_semantics/service_entry_spans/?tab=otelcollectoranddatadogexporter.