Allow selecting multiple exporters

[pellared]

Fixes #1755

Changes

• Allow selecting multiple exporters via OTEL_TRACES_EXPORTER and OTEL_METRICS_EXPORTER by using a comma-separated list.

The exporter selection environmental variables allow only to select a SINGLE one. It would be handy to allow selecting more than one e.g. otlp,console.

[carlosalberto]

Maybe totally straightforward, but I'd like to see a prototype of this in any language.

[SergeyKanzhelev]

I think the scenario in PR description otlp,console totally makes a good use cases. In production cases there will be situations when different exporters will need different processors - like a sampling or different aggregations. I suggest we stop at multiple exporters for now in environment variables.

@carlosalberto has a good suggestion. I think it will be trivial to implement in .NET, can you please demonstrate the implementation - issue open-telemetry/opentelemetry-dotnet#1453?

[pellared]

@carlosalberto @SergeyKanzhelev

Maybe totally straightforward, but I'd like to see a prototype of this in any language.

I think it will be trivial to implement in .NET, can you please demonstrate the implementation - issue open-telemetry/opentelemetry-dotnet#1453?

I do not think it is trivial. Especially in .NET as it has almost no support for env vars at the moment. References:

• https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#environment-variables
• SDK configuration from ENV Variable opentelemetry-dotnet#1453 (comment)

As a rule of thumb, I feel that the design should be similar to OTEL_PROPAGATORS. I made a quick look at the Python SDK. The OTEL_PROPAGATORS implementation looks nice and probably a similar pattern could be used for exporters. WDYT @owais?

I have no idea what is the ETA for implementing open-telemetry/opentelemetry-dotnet#1453. However, I see no problem having this PR open until someone actually has time to implement it (in whatever language) 😉

[jkwatson]

For what it's worth, we implemented this for trace exporters in Java before the env var spec got finalized, at which point we pulled it back to a single exporter. It's a little tricky, however, since just specifying an exporter doesn't determine what span processor you might want to use. And, for a logging exporter, a batch span processor probably doesn't make much sense.

In the otel java instrumentation agent, we automatically add the logging exporter if the user enables debug mode for the javaagent. And, in that case, we wire it up with a SimpleSpanProcessor, as that makes more sense for the logging use-case.

[owais]

Maybe totally straightforward, but I'd like to see a prototype of this in any language.

Python already supports this exactly as proposed and it works quite well.

https://github.com/open-telemetry/opentelemetry-python/blob/b3455cd1164f9c5f336cc26a52fb351cb422b0b2/opentelemetry-distro/src/opentelemetry/distro/__init__.py#L78-L82
https://github.com/open-telemetry/opentelemetry-python/blob/b3455cd1164f9c5f336cc26a52fb351cb422b0b2/opentelemetry-distro/src/opentelemetry/distro/__init__.py#L141-L146
https://github.com/open-telemetry/opentelemetry-python/blob/b3455cd1164f9c5f336cc26a52fb351cb422b0b2/opentelemetry-distro/src/opentelemetry/distro/__init__.py#L48

[SergeyKanzhelev]

@owais what's the behavior of exporters w.r.t. @jkwatson's comment? Will console exporter will be added with the batch processor? Is there anything worth noting in this PR?

[owais]

@SergeyKanzhelev @jkwatson Python implement does not treat console or any other exporter as a special case. It always uses batch exporter. Ability to quickly add console exporter without removing a real exporter is already immensely useful even if the output is a bit delayed because of batching. I have not seen this cause any issues in practice. In fact it actually is quite nice since you can tell exactly when a remote exporter is supposed to export the spans.

Besides, console exporter is not the only use case as this can be used to export to multiple remote exporters as well.

IMO trying to specify multiple combinations of process+exporter via env vars will get too complex and we should expect users to setup such pipelines with code instead.

[carlosalberto]

From the Spec call yesterday: although supporting multiple tracer exporters is recurrent request, supporting multiple metrics exporters is not a clear thing (e.g. pull vs push based systems).

Moreover, this is closely related to the effort by the new Instrumentation group/SIG so let's hold this a little bit for now, while we wait for such group to review it.

[pellared]

@carlosalberto

From the Spec call yesterday: although supporting multiple tracer exporters is recurrent request, supporting multiple metrics exporters is not a clear thing (e.g. pull vs push based systems).

Is there any problem to make them working "side-by-side"? I think that each concrete exporter should have its own set of environmental variables to configure it.

[jkwatson]

@carlosalberto

From the Spec call yesterday: although supporting multiple tracer exporters is recurrent request, supporting multiple metrics exporters is not a clear thing (e.g. pull vs push based systems).

Is there any problem to make them working "side-by-side"? I think that each concrete exporter should have its own set of environmental variables to configure it.

It might not be a problem eventually, but it's certainly not something that is spec'd yet for the metrics SDK, so we shouldn't be providing any configuration for it, IMO.

[SergeyKanzhelev]

@carlosalberto suggested to wait for instrumentation group to pick this PR up. Thinking about it, this is a small adjustment that doesn't solve problems like different processors and things like this. Just a convenience improvement. So I'd vote to just accept this proposal. Unlikely it will block any sophisticated solution on telemetry pipelines configuration that instrumentation group will come up with. Thoughts?

[jmacd]

@SergeyKanzhelev and @carlosalberto I'm not sure how this belongs with an instrumentation group. Isn't this about SDK configuration? I would assume that many defaults will apply if I set OTLP_METRICS_EXPORTER=otlp,prometheus and that essential configuration items might get dedicated environment variables (e.g., OTLP_METRICS_PROMETHEUS_PORT, etc.)

[SergeyKanzhelev]

I don't have any more concerns and @carlosalberto didn't repond for a while, I assume it's ok. Merging.