A way to export internal data as OTLP proto format

[chenzhihao]

Is your feature request related to a problem? Please describe.

We are trying to add a customised OPTL collector Exporter to send the metrics and spans via HTTP1.1 to our ingestion service by using OPTL protobuf binary as the transport data format.

But we can't simply convert the pdata.Traces and pdata.Metrics to the OPTL Protobuf-based structs as these structs are in the private fields of pdata in the implementation of OPTL collector:

type Traces struct {
	orig *[]*otlptrace.ResourceSpans
}

type MetricData struct {
	orig *[]*otlpmetrics.ResourceMetrics
}

Describe the solution you'd like

Create an HTTP equivalent of opentelemetry-collector/exporter/otlpexporter and opentelemetry-collector/exporter/otlpexporter which export the spans and metrics via HTTP1.1 by using OPTL protobuf binary as the transport data format.

Describe alternatives you've considered

Add public APIs in OPTL collector which can map the internal data to OPTL protobuf-based structs.
e.g. a public helper function as func MetricDataToOtlp(m pdata.Metrics) *otlpmetrics.ResourceMetrics {}

Additional context

We considered to follow the implementation of opentelemtry-collector-contribute to map the pdata to OpenCensus Protobuf-based structs:

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/exporter/newrelicexporter/newrelic.go#L86

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/exporter/newrelicexporter/newrelic.go#L117

But we thought this is not an ideal long term solution as eventually OpenCensus could be retired.

[bogdandrutu]

Solving partially this #1720

[chenzhihao]

@bogdandrutu Thanks for your reply!

After applying this PR we can map the internal data to []*otlpmetrics.ResourceMetrics, but the metric batch struct ExportMetricsServiceRequest is still private and can not be used by us directly: https://github.com/open-telemetry/opentelemetry-collector/blob/master/internal/data/opentelemetry-proto-gen/collector/metrics/v1/metrics_service.pb.go#L38

Could we ask you to shed some lights on how should we use the generated OPTL protobuf structs? Does that make sense to either:

• move the /internal/data to pkg/data so that we can use these protobuf structs to build the customised Collector exporter
• or use a central public repo for the generated protobuf like what OpenCesus did
• or have some public helper functions to generate the ExportMetricsServiceRequest struct

Also, will we apply a similar change for span soon?

[bogdandrutu]

There are 2 things:

1. Soon will have time to implement the otlp over http.
2. We still don't know how and if we want to expose the proto that we use internally (we may reimplement some messages for performance, etc). So looks like a decision in that sense is required as well.

Probably this week I will give you the exporter, then will spend more time thinking about the protos, as I said is not just exposing is also making sure that we don't stop our progress to improve performance

[bogdandrutu]

It may be easier to expose a ToProtoBytes method that serializes things for you as well as a ToJsonString without exposing the data structures that we use to encode/decode

[tigrannajaryan]

@chenzhihao any reason you do not want to use official OTLP/HTTP protocol?

[bogdandrutu]

I can see one: It is not implemented yet :)

[tigrannajaryan]

For reference, OTLP/HTTP exporter is planned for GA: #882 and we could use some help to implement it.

[chenzhihao]

It may be easier to expose a ToProtoBytes method that serializes things for you as well as a ToJsonString without exposing the data structures that we use to encode/decode

@bogdandrutu if we could have the helper method ToProtoBytes to generate the marshalled ExportMetricsServiceRequest, ExportTraceServiceRequest, that would be really helpful!

(If we could eventually have the exporter which can transport otlp over http that would be awesome!)

[bogdandrutu]

The exporter is coming soon. The helper methods feel free to craft a PR, I need to see how it looks before making a decision (just the signatures is enough)

[chenzhihao]

@bogdandrutu I created a PR draft for the functions: #1741

Do you think this makes sense? If we can make a decision I will go on working on it.

[chenzhihao]

I think this may give us more flexibilities for consuming these Protobuf bytes comparing to using the coming OTLP/HTTP exporter directly.

For instance, we are aiming to send these protobuf bytes via HTTP1.1 with some customized HTTP header metadata. This kind of specific requirement could be hard to be achieved by using the coming OTLP/HTTP exporter.