Bootstrap OpenTelemetry Service using OpenCensus Service as the base

[tigrannajaryan]

Goals

We need to bootstrap the OpenTelemetry Service using the existing OpenCensus Service codebase. We previously discussed with @bogdandrutu, @pjanotti and @songy23 and agreed to split the Service codebase into 2 parts: core and contrib. This bootstrapping is a good opportunity to do the splitting by only including in the OpenTelemetry Service core the minimum number of receivers and exporters and moving the rest of functionality to a contrib package (most vendor-specific code).

The contrib package and vendor-specific receivers and exporters will continue to be available and there is no intent to retire it. The intent is to have a clear decoupling in the codebase that facilitates independent contribution of new components in the future, allows to easily create customized versions of a Service and makes it clear that core contributors will be responsible for maintenance of the core while vendor-specific components will be maintained by corresponding vendors (note: this does not exclude dual participation at all - some developers will likely work for vendors and will also be core maintainers).

Proposed Approach

I suggest the following bootstrapping approach:

1. Fork OpenCensus Service repo to OpenTelemetry org (or copy all commits - it is desirable to preserve the commit history).

2. Identify receivers, exporters and processors that will be part of the core.

3. Create a new OpenTelemetry Service binary in the same repo (otel_service). This will reuse existing functionally that is already in the codebase, but will only include receivers and exporters which we consider to be part of the core.

4. The new otel_service codebase will contain improvements which are already in progress and which are aimed at making the codebase extensible and enable the splitting to core and contrib. This includes 3 initiatives:

   4.1 Decoupling of receiver and exporter implementations from the core logic.

   4.2 Introduction of receiver and exporter factories that can be individually registered to activate them.

   4.3 Implementation of the new configuration format that makes use of factories and allows for greater flexibility in the configuration.

   The functionally of the new otel_service will heavily lean on existing implementation and will be mostly a superset of the current agent/collector functionality when considering core receivers and exporters only (however we will allow deviations if it saves significant implementation effort and makes the service better).

5. Provide guidelines and example implementations for vendors to follow when they add new receivers and exporters to the contrib package.

6. With the help of vendors move remaining receivers and exporters to a contrib package (either to a separate repo or to a separate sub-directory in the main repo - TBD).

7. Provide OpenCensus-to-OpenTelemetry Service migration guidelines for end-users who want to migrate. This will include recommendations on configuration file migration. We will also consider the possibility to support old configuration format in the new binary.

This approach allows us to have significant progress towards 2 stated goals in our vision document: unify the codebase for agent and collector and make the service more extensible.

I would be glad to hear everyone's opinion on this proposal and I am happy to take the lead on implementing the plan once it is accepted.

[pjanotti]

2. Identify receivers, exporters and processors that will be part of the core.

Proposed list of receivers:

• Prometheus
• Jaeger (agent and collector ones)
• Zipkin
• OpenCensus, temporarily until OpenTelemetry one is available (we may want to keep OC for longer to facilitate migrations)

We should not bring any of the OpenCensus exporters based on the client library: we experienced issues regarding inconsistency buffering, retries, observability, etc. We should aim to have the selected exporters implemented from 3 separate components:

1. Format translators: from internal format (we can start with OC since we don't have OTel ready yet)
2. A generic queue with consistent observability and configuration options (e.g.: batching) for all exporters
3. A sender/emitter in charge of simply sending a single transformed data batch to its respective backend

This is much in line with a "full SDK" for different data formats, this does not prevent alternate implementations while making a low bar for writing observable/consistent custom exporters. We can work closely with the golang SIG to ensure that these interfaces converge and writing a lib exporter also gives you the ability to use the same code to have an export on OCService contrib.

"... will be mostly a superset of the current agent/collector..."

Should be "... will be mostly a subset of the current agent/collector...". The idea being that instead of having those receivers/exporters working from day one and then moving them to contrib, we instead start with the targeted core support, removing the non-core right after the initial repo migration. Any operational gap should be filled via some combination of OCService + OTelService (item 7 on the list above). Going this route vendors can work and retire OCService at the pace appropriate to them without forcing us to start with a large legacy (it lives on OC repo).

[tigrannajaryan]

Done, added as a doc.