Customizing OpenTelemetry .NET SDK for Tracing

TracerProvider

As shown in the Getting Started - ASP.NET Core Application and Getting Started - Console Application docs, OpenTelemetry tracing is managed by a TracerProvider instance configured using the TracerProviderBuilder API.

TracerProviderBuilder exposes various methods to configure the provider (ex: SetSampler, AddProcessor, etc.) which are explained in subsequent sections of this document. It is also common for library authors to target TracerProviderBuilder for extension methods which help configure SDK plug-in components.

Building a TracerProvider

There are two different ways to create a TracerProviderBuilder.

Using OpenTelemetry.Extensions.Hosting

For ASP.NET Core and .NET Generic host users, helper extensions are provided in the OpenTelemetry.Extensions.Hosting package to simplify configuration and management of the TracerProvider.

using OpenTelemetry.Trace;

var appBuilder = WebApplication.CreateBuilder(args);

appBuilder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder.AddConsoleExporter());

Note

The AddOpenTelemetry extension automatically starts and stops the TracerProvider with the host.

Using Sdk.CreateTracerProviderBuilder

Sdk.CreateTracerProviderBuilder() is provided on all runtimes to create TracerProviders when either hosting is not available or multiple providers are required.

Call Sdk.CreateTracerProviderBuilder() to obtain a builder and then call Build() once configuration is done to retrieve the TracerProvider instance.

Note

Once built changes to TracerProvider configuration are not allowed, with the exception of adding more processors.

In most cases a single TracerProvider is created at the application startup, and is disposed when application shuts down.

The snippet below shows how to build a basic TracerProvider and dispose it at the end of the application. This will create a provider with default configuration, and is not particularly useful. The subsequent sections shows how to build a more useful provider.

using OpenTelemetry;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder().Build();
// ....

// Dispose at application shutdown
tracerProvider.Dispose()

TracerProvider configuration

TracerProvider holds the tracing configuration, which includes the following:

The list of ActivitySources (aka Tracers) from which traces are collected.
The list of instrumentations enabled via InstrumentationLibrary.
The list of Processors, including exporting processors which exports traces to Exporters
The Resource associated with the traces.
The Sampler to be used.

Activity Source

ActivitySource denotes a Tracer, which is used to create activities. The SDK follows an explicit opt-in model for listening to activity sources. i.e, by default, it listens to no sources. Every activity source which produce telemetry must be explicitly added to the tracer provider to start collecting traces from them.

AddSource method on TracerProviderBuilder can be used to add a ActivitySource to the provider. The name of the ActivitySource (case-insensitive) must be the argument to this method. Multiple AddSource can be called to add more than one source. It also supports wildcard subscription model as well.

It is not possible to add sources after the provider is built, by calling the Build() method on the TracerProviderBuilder.

The snippet below shows how to add activity sources to the provider.

using OpenTelemetry;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    // The following subscribes to activities from Activity Source
    // named "MyCompany.MyProduct.MyLibrary" only.
    .AddSource("MyCompany.MyProduct.MyLibrary")
    // The following subscribes to activities from all Activity Sources
    // whose name starts with  "AbcCompany.XyzProduct.".
    .AddSource("AbcCompany.XyzProduct.*")
    .Build();

See Program.cs for complete example.

Note

A common mistake while configuring TracerProvider is forgetting to add all ActivitySources to the provider. It is recommended to leverage the wild card subscription model where it makes sense. For example, if your application is expecting to enable tracing from a number of libraries from a company "Abc", the you can use AddSource("Abc.*") to enable all sources whose name starts with "Abc.".

Instrumentation

While the OpenTelemetry API can be used to instrument any library manually, Instrumentation Libraries are available for a lot of commonly used libraries. Such instrumentations can be added to the TracerProvider. It is not required to attach the instrumentation to the provider, unless the life cycle of the instrumentation must be managed by the provider. If the instrumentation must be activated/shutdown/disposed along with the provider, then the instrumentation must be added to the provider.

Typically, the instrumentation libraries provide extension methods on TracerProviderBuilder to allow adding them to the TracerProvider. Please refer to corresponding documentation of the instrumentation library to know the exact method name.

Follow this document to learn about the instrumentation libraries shipped from this repo and writing custom instrumentation libraries.

Processors & Exporters

Processors expose hooks for start and end processing of Activity instances. If no processors are configured then traces are simply dropped by the SDK. The AddProcessor method on TracerProviderBuilder is provided to add a processor to the SDK pipeline. There can be any number of processors added to the provider and they are invoked in the same order as they are added. Unlike Sampler and Resource, processors can be added to the provider even after it is built.

Exporters expose hooks for exporting batches of completed Activity instances (a batch may contain a single or many records) and are called by processors. Two base processor classes SimpleExportProcessor & BatchExportProcessor are provided to support invoking exporters through the processor pipeline and implement the standard behaviors prescribed by the OpenTelemetry specification.

Note

The SDK only ever invokes processors and has no direct knowledge of any registered exporters.

Processor Configuration

The snippet below shows how to add processors to the provider before and after it is built.

using OpenTelemetry;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddProcessor(new MyProcessor1())
    .AddProcessor(new MyProcessor2()))
    .Build();

// Processors can be added to provider even after it is built.
// Only those traces which are emitted after this line, will be sent to it.
tracerProvider.AddProcessor(new MyProcessor3());

Note

The order of processor registration is important. Each processor added is invoked in order by the SDK. For example if a simple exporting processor is added before an enrichment processor the exported data will not contain anything added by the enrichment because it happens after the export.

Note

A TracerProvider assumes ownership of all processors added to it. This means that the provider will call the Shutdown method on all registered processors when it is shutting down and call the Dispose method on all registered processors when it is disposed. If multiple providers are being set up in an application then separate instances of processors MUST be registered on each provider. Otherwise shutting down one provider will cause the shared processor(s) in other providers to be shut down as well which may lead to undesired results.

Processors can be used for enriching, exporting, and/or filtering telemetry.

To enrich telemetry, users may write custom processors overriding the OnStart and/or OnEnd methods (as needed) to implement custom logic to change the data before it is passed to the next processor in the pipeline.

For exporting purposes, the SDK provides the following built-in processors:

BatchExportProcessor<T> : This is an exporting processor which batches the telemetry before sending to the configured exporter.

The following environment variables can be used to override the default values of the BatchExportActivityProcessorOptions.

Environment variable	`BatchExportActivityProcessorOptions` property
`OTEL_BSP_SCHEDULE_DELAY`	`ScheduledDelayMilliseconds`
`OTEL_BSP_EXPORT_TIMEOUT`	`ExporterTimeoutMilliseconds`
`OTEL_BSP_MAX_QUEUE_SIZE`	`MaxQueueSize`
`OTEL_BSP_MAX_EXPORT_BATCH_SIZE`	`MaxExportBatchSizeEnvVarKey`

SimpleExportProcessor<T> : This is an exporting processor which passes telemetry to the configured exporter immediately without any batching.

Note

A special processor CompositeProcessor<T> is used by the SDK to chain multiple processors together and may be used as needed by users to define sub-pipelines.

Note

The processors shipped from this SDK are generic implementations and support tracing and logging by implementing Activity and LogRecord respectively.

Follow this document to learn about writing custom processors.

Exporter Configuration

The snippet below shows how to add export processors to the provider before it is built.

using OpenTelemetry;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddProcessor(new BatchActivityExportProcessor(new MyExporter1()))
    .AddProcessor(new SimpleActivityExportProcessor(new MyExporter2()))
    .Build();

It is also common for exporters to provide their own extensions to simplify registration. The snippet below shows how to add the OtlpExporter to the provider before it is built.

using OpenTelemetry;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
   .AddOtlpExporter()
   .Build();

Follow this document to learn about writing custom exporters.

Resource

Resource is the immutable representation of the entity producing the telemetry. If no Resource is explicitly configured, the default is to use a resource indicating this Service and Telemetry SDK. The ConfigureResource method on TracerProviderBuilder can be used to configure the resource on the provider. ConfigureResource accepts an Action to configure the ResourceBuilder. Multiple calls to ConfigureResource can be made. When the provider is built, it builds the final Resource combining all the ConfigureResource calls. There can only be a single Resource associated with a provider. It is not possible to change the resource builder after the provider is built, by calling the Build() method on the TracerProviderBuilder.

ResourceBuilder offers various methods to construct resource comprising of attributes from various sources. For example, AddService() adds Service resource. AddAttributes can be used to add any additional attribute to the Resource. It also allows adding ResourceDetectors.

It is recommended to model attributes that are static throughout the lifetime of the process as Resources, instead of adding them as attributes(tags) on each Activity.

Follow this document to learn about writing custom resource detectors.

The snippet below shows configuring the Resource associated with the provider.

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(r => r.AddAttributes(new List<KeyValuePair<string, object>>
                {
                    new KeyValuePair<string, object>("static-attribute1", "v1"),
                    new KeyValuePair<string, object>("static-attribute2", "v2"),
                }))
    .ConfigureResource(resourceBuilder => resourceBuilder.AddService("service-name"))
    .Build();

It is also possible to configure the Resource by using following environmental variables:

Environment variable	Description
`OTEL_RESOURCE_ATTRIBUTES`	Key-value pairs to be used as resource attributes. See the Resource SDK specification for more details.
`OTEL_SERVICE_NAME`	Sets the value of the `service.name` resource attribute. If `service.name` is also provided in `OTEL_RESOURCE_ATTRIBUTES`, then `OTEL_SERVICE_NAME` takes precedence.

Samplers

Samplers are used to control the noise and overhead introduced by OpenTelemetry by reducing the number of samples of traces collected and sent to the processors. If no sampler is explicitly configured, the default is to use ParentBased(root=AlwaysOn). SetSampler method on TracerProviderBuilder can be used to set sampler. Only one sampler can be associated with a provider. If SetSampler is called multiple times, the last one wins. Also, it is not possible to change the sampler after the provider is built, by calling the Build() method on the TracerProviderBuilder.

The snippet below shows configuring a custom sampler to the provider.

using OpenTelemetry;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetSampler(new TraceIdRatioBasedSampler(0.25))
    .Build();

If using 1.8.0-rc.1 or newer it is also possible to configure the sampler by using the following environmental variables:

Environment variable	Description
`OTEL_TRACES_SAMPLER`	Sampler to be used for traces. See the General SDK Configuration specification for more details.
`OTEL_TRACES_SAMPLER_ARG`	String value to be used as the sampler argument.

The supported values for OTEL_TRACES_SAMPLER are:

always_off
always_on
traceidratio
parentbased_always_on,
parentbased_always_off
parentbased_traceidratio

The options traceidratio and parentbased_traceidratio may have the sampler probability configured via the OTEL_TRACES_SAMPLER_ARG environment variable.

Follow this document to learn about writing custom samplers.

Context Propagation

The OpenTelemetry API exposes a method to obtain the default propagator which is no-op, by default. This SDK replaces the no-op with a composite propagator containing the Baggage Propagator and TraceContext propagator. This default propagator can be overridden with the below snippet.

using OpenTelemetry;

Sdk.SetDefaultTextMapPropagator(new MyCustomPropagator());

Dependency injection support

Note

This information applies to the OpenTelemetry SDK version 1.4.0 and newer only.

The SDK implementation of TracerProviderBuilder is backed by an IServiceCollection and supports a wide range of APIs to enable what is generally known as dependency injection.

Dependency injection examples

For the below examples imagine a processor with this constructor:

public class MyCustomProcessor : BaseProcessor<Activity>
{
    public MyCustomProcessor(MyCustomService myCustomService)
    {
        // Implementation not important
    }
}

We want to inject MyCustomService dependency into our MyCustomProcessor instance.

Using Sdk.CreateTracerProviderBuilder()

To register MyCustomProcessor and MyCustomService we can use the ConfigureServices and AddProcessor methods:

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .ConfigureServices(services =>
    {
        services.AddSingleton<MyCustomService>();
    })
    .AddProcessor<MyCustomProcessor>()
    .Build();

When using the Sdk.CreateTracerProviderBuilder method the TracerProvider owns its own IServiceCollection. It will only be able to see services registered into that collection.

Note

It is important to correctly manage the lifecycle of the TracerProvider. See Building a TracerProvider for details.

Using the OpenTelemetry.Extensions.Hosting package

Note

If you are authoring an ASP.NET Core application or using the .NET Generic Host the OpenTelemetry.Extensions.Hosting package is the recommended mechanism.

var appBuilder = WebApplication.CreateBuilder(args);

appBuilder.Services.AddSingleton<MyCustomService>();

appBuilder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddProcessor<MyCustomProcessor>());

When using the AddOpenTelemetry & WithTracing extension methods the TracerProvider does not own its IServiceCollection and instead registers into an existing collection (typically the collection used is the one managed by the application host). The TracerProviderBuilder will be able to access all services registered into that collection. For lifecycle management, the AddOpenTelemetry registers an IHostedService which is used to automatically start the TracerProvider when the host starts and the host will automatically shutdown and dispose the TracerProvider when it is shutdown.

Note

Multiple calls to WithTracing will configure the same TracerProvider. Only a single TraceProvider may exist in an IServiceCollection \ IServiceProvider.

Dependency injection TracerProviderBuilder extension method reference

AddInstrumentation<T>: Adds instrumentation of type T into the TracerProvider.
AddInstrumentation<T>(Func<IServiceProvider, T> instrumentationFactory): Adds instrumentation of type T into the TracerProvider using a factory function to create the instrumentation instance.
AddProcessor<T>: Adds a processor of type T (must derive from BaseProcessor<Activity>) into the TracerProvider.
AddProcessor(Func<IServiceProvider, BaseProcessor<Activity>> implementationFactory): Adds a processor into the TracerProvider using a factory function to create the processor instance.
ConfigureServices: Registers a callback function for configuring the IServiceCollection used by the TracerProviderBuilder.

[!NOTE] ConfigureServices may only be called before the IServiceProvider has been created after which point services can no longer be added.
SetSampler<T>: Register type T (must derive from Sampler) as the sampler for the TracerProvider.
SetSampler(Func<IServiceProvider, Sampler> implementationFactory): Adds a sampler into the TracerProvider using a factory function to create the sampler instance.

Note

The factory functions accepting IServiceProvider may always be used regardless of how the SDK is initialized. When using an external service collection (ex: appBuilder.Services.AddOpenTelemetry()), as is common in ASP.NET Core hosts, the IServiceProvider will be the instance shared and managed by the host. When using "Sdk.Create" functions, as is common in .NET Framework hosts, the provider creates its own IServiceCollection and will build an IServiceProvider from it to make available to extensions.

Configuration files and environment variables

Note

This information applies to the OpenTelemetry SDK version 1.4.0 and newer only.

The OpenTelemetry .NET SDK integrates with the standard configuration and options patterns provided by .NET. The configuration pattern supports building a composited view of settings from external sources and the options pattern helps use those settings to configure features by binding to simple classes.

How to set up configuration

The following sections describe how to set up configuration based on the host and OpenTelemetry API being used.

Using .NET hosts with the OpenTelemetry.Extensions.Hosting package

ASP.NET Core and .NET Generic host users using the OpenTelemetry.Extensions.Hosting package do not need to do anything extra to enable IConfiguration support. The OpenTelemetry SDK will automatically use whatever IConfiguration has been supplied by the host. The host by default will load environment variables, command-line arguments, and config files. See Configuration in .NET for details.

Using Sdk.CreateTracerProviderBuilder directly

By default the Sdk.CreateTracerProviderBuilder API will create an IConfiguration from environment variables. The following example shows how to customize the IConfiguration used by Sdk.CreateTracerProviderBuilder for cases where additional sources beyond environment variables are required.

// Build configuration from sources. Order is important.
var configuration = new ConfigurationBuilder()
    .AddJsonFile("./myOTelSettings.json")
    .AddEnvironmentVariables()
    .AddCommandLine(args)
    .Build();

// Set up a TracerProvider using the configuration.
var provider = Sdk.CreateTracerProviderBuilder()
    .ConfigureServices(services => services.AddSingleton<IConfiguration>(configuration))
    .Build();

OpenTelemetry Specification environment variables

The OpenTelemetry Specification defines specific environment variables which may be used to configure SDK implementations.

The OpenTelemetry .NET SDK will look for the environment variables defined in the specification using IConfiguration which means in addition to environment variables users may also manage these settings via the command-line, configuration files, or any other source registered with the .NET configuration engine. This provides greater flexibility than what the specification defines.

Note

Not all of the environment variables defined in the specification are supported. Consult the individual project README files for details on specific environment variable support.

As an example the OpenTelemetry Specification defines the OTEL_SERVICE_NAME environment variable which may be used to configure the service name emitted on telemetry by the SDK.

A traditional environment variable is set using a command like set OTEL_SERVICE_NAME=MyService on Windows or export OTEL_SERVICE_NAME=MyService on Linux.

That works as expected but the OpenTelemetry .NET SDK is actually looking for the OTEL_SERVICE_NAME key in IConfiguration which means it may also be configured in any configuration source registered with the IConfigurationBuilder used to create the final configuration for the host.

Below are two examples of configuring the OTEL_SERVICE_NAME setting beyond environment variables.

Using appsettings.json:
```
{
   "OTEL_SERVICE_NAME": "MyService"
}
```

Using command-line:

dotnet run --OTEL_SERVICE_NAME "MyService"

Note

The .NET Configuration pattern is hierarchical meaning the order of registered configuration sources controls which value will seen by the SDK when it is defined in multiple sources.

Using the .NET Options pattern to configure the SDK

Options are typically simple classes containing only properties with public "getters" and "setters" (aka POCOs) and have "Options" at the end of the class name. These options classes are primarily used when interacting with the TracerProviderBuilder to control settings and features of the different SDK components.

Options classes can always be configured through code but users typically want to control key settings through configuration.

The following example shows how to configure OtlpExporterOptions by binding to an IConfiguration section.

Json config file (usually appsettings.json):

{
  "OpenTelemetry": {
    "Otlp": {
      "Endpoint": "http://localhost:4317"
    }
  }
}

Code:

var appBuilder = WebApplication.CreateBuilder(args);

appBuilder.Services.Configure<OtlpExporterOptions>(
    appBuilder.Configuration.GetSection("OpenTelemetry:Otlp"));

appBuilder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder.AddOtlpExporter());

The OpenTelemetry .NET SDK supports running multiple TracerProviders inside the same application and it also supports registering multiple similar components such as exporters into a single TracerProvider. In order to allow users to target configuration at specific components a "name" parameter is typically supported on configuration extensions to control the options instance used for the component being registered.

The below example shows how to configure two OtlpExporter instances inside a single TracerProvider sending to different ports.

Json config file (usually appsettings.json):

{
  "OpenTelemetry": {
    "OtlpPrimary": {
      "Endpoint": "http://localhost:4317"
    },
    "OtlpSecondary": {
      "Endpoint": "http://localhost:4327"
    },
  }
}

Code:

var appBuilder = WebApplication.CreateBuilder(args);

appBuilder.Services.Configure<OtlpExporterOptions>(
    "OtlpPrimary",
    appBuilder.Configuration.GetSection("OpenTelemetry:OtlpPrimary"));

appBuilder.Services.Configure<OtlpExporterOptions>(
    "OtlpSecondary",
    appBuilder.Configuration.GetSection("OpenTelemetry:OtlpSecondary"));

appBuilder.Services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddOtlpExporter(name: "OtlpPrimary", configure: null)
        .AddOtlpExporter(name: "OtlpSecondary", configure: null));

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Customizing OpenTelemetry .NET SDK for Tracing

TracerProvider

Building a TracerProvider

Using OpenTelemetry.Extensions.Hosting

Using Sdk.CreateTracerProviderBuilder

TracerProvider configuration

Activity Source

Instrumentation

Processors & Exporters

Processor Configuration

Exporter Configuration

Resource

Samplers

Context Propagation

Dependency injection support

Dependency injection examples

Using Sdk.CreateTracerProviderBuilder()

Using the OpenTelemetry.Extensions.Hosting package

Dependency injection TracerProviderBuilder extension method reference

Configuration files and environment variables

How to set up configuration

Using .NET hosts with the OpenTelemetry.Extensions.Hosting package

Using Sdk.CreateTracerProviderBuilder directly

OpenTelemetry Specification environment variables

Using the .NET Options pattern to configure the SDK

Files

README.md

Latest commit

History

README.md

File metadata and controls

Customizing OpenTelemetry .NET SDK for Tracing

TracerProvider

Building a TracerProvider

Using OpenTelemetry.Extensions.Hosting

Using Sdk.CreateTracerProviderBuilder

TracerProvider configuration

Activity Source

Instrumentation

Processors & Exporters

Processor Configuration

Exporter Configuration

Resource

Samplers

Context Propagation

Dependency injection support

Dependency injection examples

Using Sdk.CreateTracerProviderBuilder()

Using the OpenTelemetry.Extensions.Hosting package

Dependency injection TracerProviderBuilder extension method reference

Configuration files and environment variables

How to set up configuration

Using .NET hosts with the OpenTelemetry.Extensions.Hosting package

Using Sdk.CreateTracerProviderBuilder directly

OpenTelemetry Specification environment variables

Using the .NET Options pattern to configure the SDK