Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial requirements for integration test #630

Closed
wants to merge 11 commits into from
42 changes: 42 additions & 0 deletions specification/library-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,3 +115,45 @@ guidelines on the performance expectations that API implementations should meet,
See the [Concurrency and Thread-Safety](concurrency.md) specification for
guidelines on what concurrency safeties should API implementations provide
and how they should be documented.

### Integration Test

The goal of integration test is to make sure:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The goal of integration test is to make sure:
The goal of the integration test is to make sure:

(or plural?)

* The API package and SDK package are designed properly, so another SDK can be used to work with the API package.
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved
* The exporters are functional, and the emitted data meets expectations.
* The context propagation is implemented correctly, so a distributed trace could be built among multiple services.
* Sampling logic is implemented consistently across the services, and across the SDKs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Across which services?


A test harness is needed to verify each language library's integration and interoperability with other libraries and backends.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which other libraries? Do you mean other OpenTelemetry implementations (API + default SDK) in other processes?

carlosalberto marked this conversation as resolved.
Show resolved Hide resolved

Each language library should use the test harness as part of the CI (continuous integration).

#### Test Harness
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In OpenTracing we had a separate test harness framework that was operating purely at the API level. The implementation test had to provide certain introspection capabilities to the framework to validate the behavior (fairly limited since OT was not very prescriptive about the actual behavior). This was especially useful for dynamic languages.


The test harness should include:
* A document which explains how to use it to implement integration test.
* A [Docker compose](https://docs.docker.com/compose/).
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved
* A container image which provides integration test service that can be invoked from the library's CI job.
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved
* A simplied version of the OpenTelemetry C++ SDK, which will be used to test the interfacing between API and SDK. For example, using OpenTelemetry Python API with C++ SDK.
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved
* The latest version of OpenTelemetry Collector.
* The latest stable version of Prometheus backend.
* The latest stable version of Jaeger backend.
* The latest stable version of Zipkin backend.

#### CI Requirement

Each language library should implement a test service which interacts with the test harness. The test service should include:
* A REST endpoint which is used to receive incoming requests from the harness and other services that are involved during the CI.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clarify what you mean with "other services that are involved during the CI". Currently it sounds like this was about CI services like CircleCI or AppVeyor.

* The ability to react to the requests from the harness.
* The ability to specify the SDK.
* Without the SDK
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved
* With the language specific SDK
* With the simplifed C++ SDK
* The ability to configure the SDK, when the language specific SDK is used.
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved
* Sampler
* Exporter
* The ability to call other REST endpoints based on the instruction given by the harness.

Each language SDK should determine which platform and runtime version should be tested. For example, a language library could decide to only cover one specific combination of platform and runtime during integration test, and rely on the unit test to cover the other platforms and runtime versions.
carlosalberto marked this conversation as resolved.
Show resolved Hide resolved

### Unit Test