Support monitoring bytes read/written (compressed and uncompressed) by a pipeline

[jmacd]

Is your feature request related to a problem? Please describe.

It is difficult to observe how many bytes are sent and received from collector pipelines. As an example of this test, I've examined the OTel-Collector-Contrib testbed package which has tests for both OTLP-gRPC and OTLP-gRPC-gzip. However, since "gzip" is the default compression strategy, these tests are actually identical in a way that cannot easily be observed.

Describe the solution you'd like

I would like to extend the obsreport interfaces to allow pipeline components to annotate how many bytes were read or written as a result of the individual pipeline item. I would make the new interface optional, so that not all components have to be upgraded to report these counts, if they don't make sense.

I would like the testbed to snoop these metrics for the system under test and incorporate them in its reports. This would allow us to evaluate the cpu/memory/network implications of different exporter / receiver configurations.

I would like to be able to monitor both compressed and uncompressed data size, ideally.

Describe alternatives you've considered

The testbed could manually try to calculate the amount of network traffic. This would fix the testbed, but it would leave us in a state where the collector is still not observable. For us to monitor collector in production we need metrics.

It's not actually required to extend obsreport if the benefit is only needed for OTLP components. I would be mostly happy with added metrics to the OTLP receiver and exporter, however I think any protocol being used in production will benefit from being monitored in terms of bytes transported.

The use of host-level interface-level metrics could potentially be used as a proxy for specific pipeline metrics, but I would not prefer this solution. The use of gRPC interceptors through the otel-go-contrib instrumentation/google.golang.org/grpc/otelgrpc instrumentation package.

[bogdandrutu]

In OpenCensus this was done by the http/grpc metrics. Do we need to monitor this in obsreport, or the integration that otel-go will provide should do that for us?

[jmacd]

@bogdandrutu That's a good question--I thought I'd file a WIP PR to show how it looks without using third-party instrumentation for the network stats. I do think that's an option, though currently otel-go-contrib has only gRPC interceptors for tracing, not gRPC stats handlers for metrics. I imagine that an obsreport-builtin feature will make this easier to use in other protocols and more consistent for observing collectors. I imagine this would be easier to use in territory where there is not an existing instrumentation library, where the receiver or exporter might be using a low-level network interface.

Anyway, even in the draft PR #6712 you can see I decided to duplicate the 20-or-so lines of code that are needed to be a gRPC stats handler, even though it's identical in the OTLP-gRPC exporter and OTLP-gRPC receiver case. If there was a go-contrib package that contained those 20 lines of code, I'd be happy, but I also want to move the collector forward and not block on otel-go instrumentation. PTAL at #6712. Separately, I will post a draft integration PR for the otel-collector-contrib testbed as described above.

[jmacd]

@bogdandrutu Would you please take a look at #6712 and offer some guidance on whether you think this feature is worth supporting? I think the community is motivated to see the OTel-Collector-Contrib testbed remain real and viable, but honestly I think without this support built in to every exporter/receiver the testbed is sadly incomplete. I am willing to push this all the way through the testbed (as seen in open-telemetry/opentelemetry-collector-contrib#16835) but would like this support to be agreed on before I proceed.