You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
OpenTelemetry Collector Contrib's groupbyattrs processor can be also used for compaction (essentially putting records with matching Instrumentation Scope/Resource under single Scope/Resource)
Such capability is useful in at least two cases:
General optimization of data representation
Even with using batching, records that have a matching Instrumentation Scope and Resource can be spread across separate ResourceSpans/ResourceLogs/ResourceMetrics. This might happen due to a number of reasons - e.g. having small input batches or due to fragmentation created by groupbytrace processor or loadbalancing exporter.
By "compacting" data together using groupbyattrs processor, we avoid duplication of Attributes under several Resource objects and data is represented more efficiently
Reduction of export requests for Jaeger exporter
Due to specific of Jaeger proto, the exporter creates a request for each ResourceSpans. If data comes fragmented, this might lead to a significant increase in number of requests which due to connection latency might further lead to dropping data. An example of that is discussed in opentelemetry-collector-contrib/#9009. For explanation on how groupbyattrs processor helps here, refer to this comment
Proposal
Perhaps the code used for grouping (and compaction) could be put into some opentelemetry-collector package and then compaction could be used in exporterhelper together with batching, as discussed in #4646
Alternative
If we conclude this problem is Jaeger specific, I would suggest to move the common code to some package in opentelemetry-collector-contrib and then use it in groupbyattrs processor and jaeger exporter
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
OpenTelemetry Collector Contrib's groupbyattrs processor can be also used for compaction (essentially putting records with matching Instrumentation Scope/Resource under single Scope/Resource)
Such capability is useful in at least two cases:
General optimization of data representation
Even with using batching, records that have a matching Instrumentation Scope and Resource can be spread across separate ResourceSpans/ResourceLogs/ResourceMetrics. This might happen due to a number of reasons - e.g. having small input batches or due to fragmentation created by
groupbytrace
processor orloadbalancing
exporter.By "compacting" data together using
groupbyattrs
processor, we avoid duplication of Attributes under several Resource objects and data is represented more efficientlyReduction of export requests for Jaeger exporter
Due to specific of Jaeger proto, the exporter creates a request for each ResourceSpans. If data comes fragmented, this might lead to a significant increase in number of requests which due to connection latency might further lead to dropping data. An example of that is discussed in opentelemetry-collector-contrib/#9009. For explanation on how
groupbyattrs
processor helps here, refer to this commentProposal
Perhaps the code used for grouping (and compaction) could be put into some opentelemetry-collector package and then compaction could be used in
exporterhelper
together with batching, as discussed in #4646Alternative
If we conclude this problem is Jaeger specific, I would suggest to move the common code to some package in
opentelemetry-collector-contrib
and then use it ingroupbyattrs
processor andjaeger
exporterThe text was updated successfully, but these errors were encountered: