Skip to content

Commit

Permalink
Add draft logging library SDK specification
Browse files Browse the repository at this point in the history
This is the first cut from the [OTEP0150](https://github.com/open-telemetry/oteps/blob/main/text/logs/0150-logging-library-sdk.md)
This a draft that requires cleanup and clarifications that will be done in follow-up PR.

This PR primarily copies the OTEP0150 content, minus sections which are typically
only present in OTEPS (Motivaiton, Alternates, Future Possibilities, etc).
  • Loading branch information
tigrannajaryan committed Feb 9, 2022
1 parent ec2e118 commit ad2ca4a
Show file tree
Hide file tree
Showing 5 changed files with 204 additions and 0 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
204 changes: 204 additions & 0 deletions specification/logs/logging-library-sdk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# Logging Library SDK Specification

**Status**: [Experimental](../document-status.md)

This document contains the first cut from the
[OTEP0150](https://github.com/open-telemetry/oteps/blob/main/text/logs/0150-logging-library-sdk.md)
This a draft that requires cleanup and clarifications.

This specification defines how the OpenTelemetry Logging Library SDK exposes its
functionality to authors of extensions to language-specific 3rd party logging
libraries and to end users that want to produce logs in the
[OpenTelemetry manner](overview.md).

The specification defines SDK elements that to some extent mirror the
OpenTelemetry [Trace SDK](../trace/sdk.md). This ensures uniformity and
consistency of the OpenTelemetry specification and of the implementations across
traces and logs. For additional clarity the definitions in this document refer
to the Trace analogs where appropriate.

## Specification

Many existing logging libraries have some sort of extension mechanism that
allows to customize how log records are encoded and delivered to their
destinations (for example Appender in Log4j or Core in zapcore). The
OpenTelemetry Logging Library SDK is intended to be used by such extensions to
emit logs in OpenTelemetry formats.

Note: The functionality that this document describes is an SDK package. A
logging-related API package may be added in the future if we decide to have an
end-user callable logging API. Until then the functions and methods callable
from this SDK package are intended to be used by Logging Libraries only and are
NOT intended to be used by the end user and will NOT be exposed in the
OpenTelemetry API package.

Logging Library SDK is comprised of the following.

### LogEmitterProvider

Methods:

- Get LogEmitter. Accepts the instrumentation scope name and version and returns
a LogEmitter associated with the instrumentation scope.
- Shutdown.
- ForceFlush.

LogEmitterProvider can be configured at startup time, to be associated with a
Resource and with LogProcessor/LogExporter pipeline.

### LogEmitter

Methods:

- Emit(LogRecord). Emits a log record. The LogRecord and the Resource and
Instrumentation library associated with the LogEmitter will be converted into
a readable LogData and will be pushed through the SDK and the configured
LogProcessors and LogExporter. It is expected that the caller will populate
trace context related fields (TraceId,SpanId,TraceFlags) if applicable before
making the call. Open Question: do we need to also pass the Baggage so that
log processors and exporters can use it if they see the need?

Note: some languages may opt to avoid having a LogRecord data type and instead
use a more idiomatic builder pattern to prepare and emit a log record (see
e.g.
[Java discussion](https://github.com/open-telemetry/opentelemetry-java/pull/3759#discussion_r738019425))

- Flush.

### LogRecord

See LogRecord [data model](data-model.md) for the list of fields.

### LogProcessor

Plugin interface. Analog of SpanProcessor. Interface to hook the log record
emitting action.

Methods:

- Emit(LogData). Call when a log record is ready to be processed and exported.
- Shutdown.
- ForceFlush.

Built-in implementations: SimpleLogProcessor, BatchLogProcessor.

### LogData

Readable LogRecord data plus associated Resource and InstrumentationLibrary.
Analog of SpanData.

### LogExporter

Plugin interface. Analog of SpanExporter. Allows to implement protocol-specific
exporters so that they can be plugged into OpenTelemetry SDK and support sending
of log data.

Methods:

- Export(batch). Exports a batch of LogData.
- Shutdown.

## Usage

### How to Create Log4J Style Appender

An Appender implementation can be used to allow emitting log records via
OpenTelemetry Logging Library exporters. This approach is typically used for
applications which are fine with changing the log transport and is
[one of the supported](overview.md#direct-to-collector) log collection
approaches.

The Appender implementation will typically acquire a LogEmitter from the global
LogEmitterProvider at startup time, then call LogEmitter.Emit for log records
received from the application.

For languages with implicit Context, the Appender may call Context API to get
the currently [active Span](../trace/api.md#context-interaction) and populate
TraceId, SpanId, TraceFlags fields of the LogRecord before emitting it. The log
library may also have an alternate way to inject the context into log records
(e.g. MDC in Log4j).

![Appender](img/logging-library-sdk/appender.png)

This same approach can be also used for example for:

- Python logging library by creating a Handler.
- Go zap logging library by implementing the Core interface. Note that since
there is no implicit Context in Go it is not possible to get and use the
active Span.

Appenders can be created in OpenTelemetry language libraries by OpenTelemetry
maintainers, or by 3rd parties for any logging library that supports a similar
extension mechanism. This specification recommends each OpenTelemetry language
library to include out-of-the-box Appender implementation for at least one most
popular logging library.

### Logging to File

One of the possible approaches to emit and collect logs that OpenTelemetry
supports is via intermediary files. When configuring the LogEmitterProvider,
OTLP File exporter should be used to write logs to a file or stdout in either
OTLP JSON or OTLP Protobuf binary format.

![Logging to File](img/logging-library-sdk/otlp-file.png)

### Logging Directly to OTLP Network Destination

The approach is the same as for logging to a file, except OTLP/gRPC or OTLP/HTTP
exporter implementation is used.

### Implicit Context Injection

When Context is implicitly available (e.g. in Java) it may be fetched by the log
library extension synchronously for every log record by calling the
OpenTelemetry Context API and injecting the span context fields into the
LogRecord before emitting it.

Some log libraries have mechanisms specifically tailored for injecting
contextual information into log records. An example of such a mechanism is Log4j
MDC. When available such mechanisms may be the preferable place to fetch the
span context and inject it into the log records, since it usually allows
fetching of the context to work correctly even when log records are emitted
asynchronously (which otherwise can result in the incorrect implicit context
being fetched.

### Explicit Context

In languages where the Context must be provided explicitly (e.g. Go) the end
user must capture the context and explicitly pass it to the logging subsystem in
order for trace context to be recorded in Log records.

Support for OpenTelemetry for logging libraries in these languages typically can
be implemented in the form of logger wrappers that can capture the context once,
when the span is created and then use the wrapped logger to execute log
statements in a normal way. The wrapper will be responsible for injecting the
captured context in the log records.

This specification does not define how exactly it is achieved since the actual
mechanism depends on the language and the particular logging library used. In
any case the wrappers are expected to make use of the Trace Context API to get
the current active span.

See
[an example](https://docs.google.com/document/d/15vR7D1x2tKd7u3zaTF0yH1WaHkUr2T4hhr7OyiZgmBg/edit#heading=h.4xuru5ljcups)
of how it can be done for zap logging library for Go.

### Custom LogExporter

LogExporter implementations can be plugged into OpenTelemetry Logging Library to
send logs via custom protocols.

OTLP/gRPC, OTLP/HTTP, OTLP/File log exporters are provided with OpenTelemetry
Logging Library out of the box.

![Custom Exporter](img/logging-library-sdk/custom-exporter.png)

### Custom LogProcessor

LogProcessor implementations can be plugged into the OpenTelemetry Logging
Library to have custom processing of logs before they are exported.

Simple and Batch processors should be provided by the OpenTelemetry Logging
Library out of the box.

![Custom Processor](img/logging-library-sdk/custom-processor.png)

0 comments on commit ad2ca4a

Please sign in to comment.