Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial requirements for integration test #630

Closed
wants to merge 11 commits into from
Closed

Add initial requirements for integration test #630

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0d56a00
add integration test requirements
reyang Jun 1, 2020
eb72f25
add OpenCensus back
reyang Jun 2, 2020
f5f0cbf
Update specification/library-guidelines.md
reyang Jun 2, 2020
3a2b1f2
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
8512cc4
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
91837e4
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
3004da9
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
aef099f
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
6fb3739
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
c3afa9a
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
eb1e4e7
Update specification/library-guidelines.md
carlosalberto Aug 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions specification/library-guidelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,3 +115,45 @@ guidelines on the performance expectations that API implementations should meet,
See the [Concurrency and Thread-Safety](concurrency.md) specification for
guidelines on what concurrency safeties should API implementations provide
and how they should be documented.

### Integration Test

The goal of integration test is to make sure:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The goal of integration test is to make sure:
The goal of the integration test is to make sure:

(or plural?)

* The API package and SDK package are designed properly, so that an SDK other than the default SDK can be used to work with the API package.
* The exporters are functional, and the emitted data meets expectations.
* The context propagation is implemented correctly, so a distributed trace could be built among multiple services.
* Sampling logic is implemented consistently across the services, and across the SDKs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Across which services?


A test harness is needed to verify each language library's integration and interoperability with other libraries and certain backends listed below.

Each language library should use the test harness as part of the CI (continuous integration).

#### Test Harness
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In OpenTracing we had a separate test harness framework that was operating purely at the API level. The implementation test had to provide certain introspection capabilities to the framework to validate the behavior (fairly limited since OT was not very prescriptive about the actual behavior). This was especially useful for dynamic languages.


The test harness should include:
* A document which explains how to use it to implement integration test.
* A [Docker compose](https://docs.docker.com/compose/) file.
* A container image which provides an integration test service that can be invoked from the library's CI job.
* A simplified version of the OpenTelemetry C++ SDK, which will be used to test the interfacing between API and SDK. For example, using the OpenTelemetry Python API with the C++ SDK.
* The latest version of OpenTelemetry Collector.
* The latest stable version of Prometheus backend.
* The latest stable version of Jaeger backend.
* The latest stable version of Zipkin backend.

#### CI Requirement

Each language library should implement a test service which interacts with the test harness. The test service should include:
* A REST endpoint which is used to receive incoming requests from the harness and other services that are involved during the CI.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clarify what you mean with "other services that are involved during the CI". Currently it sounds like this was about CI services like CircleCI or AppVeyor.

* The ability to react to the requests from the harness.
* The ability to specify the SDK.
* Without the SDK (API-only mode)
* With the language specific SDK
* With the simplifed C++ SDK
* The ability to configure the SDK, when the language specific default SDK is used.
* Sampler
* Exporter
* The ability to call other REST endpoints based on the instruction given by the harness.

Each language SDK should determine which platform and runtime version should be tested. For example, a language library could decide to only cover one specific combination of platform and runtime during integration test, and rely on the unit tests to cover the other platforms and runtime versions.

### Unit Test