Add initial requirements for integration test

[reyang]

Added initial requirements for integration test.
Removed OpenCensus exporter requirement (I guess it is no longer required?).

[yurishkuro]

There needs to be a working prototype of this solution before it is added as a requirement to the spec.

[yurishkuro]

In OpenTracing we had a separate test harness framework that was operating purely at the API level. The implementation test had to provide certain introspection capabilities to the framework to validate the behavior (fairly limited since OT was not very prescriptive about the actual behavior). This was especially useful for dynamic languages.

• https://github.com/opentracing/opentracing-javascript/blob/master/src/test/api_compatibility.ts
• https://github.com/opentracing/opentracing-go/blob/master/harness/api_checkers.go

[arminru]

Suggested change

	The goal of integration test is to make sure:
	The goal of the integration test is to make sure:

(or plural?)

[arminru]

Across which services?

[arminru]

Which other libraries? Do you mean other OpenTelemetry implementations (API + default SDK) in other processes?

[arminru]

Please clarify what you mean with "other services that are involved during the CI". Currently it sounds like this was about CI services like CircleCI or AppVeyor.

[tsloughter]

I don't understand what is meant by like "using the C++ SDK" with the Python API. This sounds way too specific to Python because I don't see how that is something that would work in most languages -- assuming it works for the Python API?

I was working on an API for a sort of integration tests by having tests run in parallel and compare results. So you'd run the tests against a Python instrumented project and against the C++ project to verify they give the same results. But it is on my other machine, I'll paste in what I was thinking of in a little bit.

[tsloughter]

Still very rough, but really quick, my idea for integration tests so far looked like:

syntax = "proto3";

import "opentelemetry/proto/common/v1/common.proto";
import "opentelemetry/proto/resource/v1/resource.proto";
import "opentelemetry/proto/trace/v1/trace.proto";

service EndToEndService {
  rpc Export(EndToEndServiceRequest) returns (EndToEndServiceResponse) {}
}

message EndToEndServiceRequest {
  repeated Step steps = 1;
}

message EndToEndServiceResponse {
}

message Step {
  oneof step {
    ForwardTo forward_to = 1;
    SetAttributes set_attributes = 2;
    AddEvents add_events = 3;
    SetStatus set_status = 4;
  }
}

message ForwardTo {
  string host = 1;
  uint32 port = 2;
  repeated Step steps = 3;
}

message SetAttributes {
  repeated opentelemetry.proto.common.v1.AttributeKeyValue attributes = 9;
}

message AddEvents {
  repeated opentelemetry.proto.trace.v1.Span.Event events = 1;
}

message SetStatus {
  opentelemetry.proto.trace.v1.Status status = 1;
}

Export might be a bad name for the method... maybe changing to Forward.

Still a lot to add. Most importantly tracestate and probably just include the existing trace_config proto as part of either the steps or the request.

But I was having a set of workers that are configured to send out the steps to the starting points and a collector that'll just support OTLP and then verify what it receives from each.

Meaning each node would configure an exporter to send to the e2e collector, or have the opentelemetry-collector receive them all and it forward to the e2e collector.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[carlosalberto]

Hey @reyang - went and applied the suggestions that seemed trivial. Do you think you will have time to iterate on the rest of the updates anytime soon?

[reyang]

Hey @reyang - went and applied the suggestions that seemed trivial. Do you think you will have time to iterate on the rest of the updates anytime soon?

Given there were asks about the proof of concept, I think it makes sense to experiment it in one SDK and come back with some real examples. I will have the integration test in OpenTelemetry .NET and come back to this PR.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[andrewhsu]

@reyang should this PR be reopened so it can be reviewed and brought in? This is on the list of desired items for otel ga.

[reyang]

@andrewhsu I will reduce the scope and send a smaller PR in order to make progress.