Testsuite for other implementations of API

[kenfinnigan]

Though OpenTelemetry provides an implementation of the API with its SDK, it would be great if there was a TCK style of test suite that could be used to verify non OpenTelemetry implementations of the API to ensure some level of compatibility and compliance.

Has anything like this been discussed?

[carlosalberto]

Hey @kenfinnigan

It was, but long time ago ;) I agree that having such TCK layer would of of great help. In OpenTracing Python we had this one, and it was pretty, pretty useful.

Probably this is something that should actually land in the Specification, as I can imagine this being done for all the supported languages ;)

[anuraaga]

/cc @awssandra We're working on some cross-language integration tests for the SDKs, is there anything we can help with here?

[kenfinnigan]

That's a good point @carlosalberto, should an issue be opened in the specification?

[jkwatson]

Is there even enough of a specification of what a non-default SDK MUST do at this point to make this valuable? IIRC, we've been focussing on describing the default SDK's behavior, rather than any arbitrary SDK's behavior.

[Oberon00]

Is there even enough of a specification of what a non-default SDK MUST do at this point

I would think so. The API spec is clearly separated from the SDK spec and any non-default SDK must also fulfill the API spec.

[jkwatson]

So, what does it mean to "fulfill" the API spec? The observable outputs of the SDK are 100% dependent on what the SDK decides to do with API calls, and I don't think that behavior is necessarily well defined. Can you give an example of something that could be tested this way today?

[kenfinnigan]

If there is behavior within the SDK that defines what is output, is that defined in the spec?

If there's no expectation about anyone every implementing the API outside the SDK, and thus the API is purely a public API documentation piece, could that be documented somewhere if it isn't already?

[jkwatson]

I believe that the intent is that alternative API implementations are possible, but I don't believe that the spec says anything, at this point, about how those alternatives should behave, or what their outputs SHOULD (or MUST) be.

[carlosalberto]

For reference, this is what we had in OT Python - as you can see, the things we had there were basic overall, but you get an idea of the (very) minimum requirements.

Other than that, having a set of requirements for the output becomes trickier, and would require way more work - for example, there could be a harness layer that creates N-Spans with a few attributes/events/links, and check the output of a supported backend, such as Zipkin or Jaeger.

[iNikem]

Other than that, having a set of requirements for the output becomes trickier, and would require way more work - for example, there could be a harness layer that creates N-Spans with a few attributes/events/links, and check the output of a supported backend, such as Zipkin or Jaeger.

Hmmm, looks like open-telemetry/opentelemetry-java-instrumentation#298 (comment)

[jkwatson]

Other than that, having a set of requirements for the output becomes trickier, and would require way more work - for example, there could be a harness layer that creates N-Spans with a few attributes/events/links, and check the output of a supported backend, such as Zipkin or Jaeger.

Hmmm, looks like open-telemetry/opentelemetry-java-instrumentation#298 (comment)

But that's just exercising the default SDK, not any arbitrary SDK, right? So, I think my concern stands...I just don't know what we can say about arbitrary SDKs...maybe one would have no discernible output, aside from some S3 bucket somewhere, for example.

[iNikem]

The same test harness can be used to exercise any SDK and verify that Collector got and understood all the data. E.g. I am going to use the same harness for testing Splunk agent as well.

[Oberon00]

I don't think being able to talk to the collector can be a generic requirement for any SDK. The no-op implementation is the minimal SDK implementation. Any really generic test suite would be a subset of the no-op testsuite.

[jkwatson]

I'm going to close this, as I still don't think it's possible to specify any outputs for an arbitrary non-standard SDK implementation of the API.