New component: Log deduplication processor

[codeboten]

The purpose and use-cases of the new component

We're working to put together a log de-duplication processor at Honeycomb and wanted to get the temperature of the community for hosting such a component in the contrib repository.

Example configuration for the component

dedupe:
  // TTL is the time-to-live for each entry in the cache. Default is 30 seconds.
  ttl: 30s
  // MaxEntries is the maximum number of entries that can be stored in the cache. Default is 1000.
  max_entries: 1000
  // AttributeNames is the list of attributes to use as the key for deduplication.
  attribute_names: []

Telemetry data types supported

logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@codeboten @MikeGoldsmith

Sponsor (optional)

No response

Additional context

No response

[atoulme]

Do you only intend to dedup on attributes? What about timestamps and log body?

[codeboten]

log bodies would be deduped by default, the attributes_names configuration would allow users to differentiate between different sources for logs they care about. i would expect timestamps would likely be thrown away as part of the deduplication w/ only the first or last timestamp reported

[BinaryFissionGames]

We have a processor with similar functionality in our agent: https://github.com/observIQ/bindplane-agent/tree/release/v1.56.0/processor/logdeduplicationprocessor

We ended up storing the first/last observed timestamp on the attributes, as well as storing the count of deduped logs as an attribute. It seems like it could be worth considering doing something similar here as well.

It would be awesome to have functionality like this here in contrib, deduping logs is very useful.

[codeboten]

@BinaryFissionGames would you consider donating the component from that repo here? Seems like it would cover a lot of functionality already

[BinaryFissionGames]

@codeboten Yeah, we'd be happy to donate the component to contrib.

[djaglowski]

I can sponsor this.

[codeboten]

Thanks @djaglowski!

[pellared]

Related spec issue: open-telemetry/opentelemetry-specification#3931

[codeboten]

@BinaryFissionGames would you like to do the initial PRs to make the component a reality or would you prefer if I did?

[BinaryFissionGames]

@codeboten I'm a bit tight on time at the moment, if you're able to get a start on it that would be awesome.

[MikeGoldsmith]

Thanks @BinaryFissionGames - I'll prepare a PR with the code from your repo.

[MikeGoldsmith]

PR created

• [processor/logdeduplicationprocessor] Add logdedupprocessor #34465

@BinaryFissionGames would be great for you review and approve on behalf of yourself & ObserveIQ.