Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] Include compaction in exporterhelper #5201

Closed
pmm-sumo opened this issue Apr 13, 2022 · 1 comment
Closed

[Proposal] Include compaction in exporterhelper #5201

pmm-sumo opened this issue Apr 13, 2022 · 1 comment
Labels

Comments

@pmm-sumo
Copy link
Contributor

pmm-sumo commented Apr 13, 2022

Is your feature request related to a problem? Please describe.

OpenTelemetry Collector Contrib's groupbyattrs processor can be also used for compaction (essentially putting records with matching Instrumentation Scope/Resource under single Scope/Resource)

Such capability is useful in at least two cases:

  1. General optimization of data representation
    Even with using batching, records that have a matching Instrumentation Scope and Resource can be spread across separate ResourceSpans/ResourceLogs/ResourceMetrics. This might happen due to a number of reasons - e.g. having small input batches or due to fragmentation created by groupbytrace processor or loadbalancing exporter.
    By "compacting" data together using groupbyattrs processor, we avoid duplication of Attributes under several Resource objects and data is represented more efficiently

  2. Reduction of export requests for Jaeger exporter
    Due to specific of Jaeger proto, the exporter creates a request for each ResourceSpans. If data comes fragmented, this might lead to a significant increase in number of requests which due to connection latency might further lead to dropping data. An example of that is discussed in opentelemetry-collector-contrib/#9009. For explanation on how groupbyattrs processor helps here, refer to this comment

Proposal

Perhaps the code used for grouping (and compaction) could be put into some opentelemetry-collector package and then compaction could be used in exporterhelper together with batching, as discussed in #4646

Alternative

If we conclude this problem is Jaeger specific, I would suggest to move the common code to some package in opentelemetry-collector-contrib and then use it in groupbyattrs processor and jaeger exporter

@pmm-sumo
Copy link
Contributor Author

Wanted to bring this issue for consideration. It might be Jaeger specific but then I'm wondering if this could be useful for other exporters as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant