[kafkareceiver] - dynamic metadata support

[VihasMakwana]

Component(s)

receiver/kafka

Is your feature request related to a problem? Please describe.

It's an enhancement request

Describe the solution you'd like

Users can specify custom metadata fields in Kafka message headers.
Currently, kafkareceiver receives the log, unmarshals it, and passes it forward.
It ignores the header field.
Users might want to attach metadata in headers and use that data in later stages.

Instead of ignoring the headers, we can also pass them on to the next stage.

Describe alternatives you've considered

Open for any other alternatives

Additional context

No response

[github-actions]

Pinging code owners:

• kafkareceiver: @pavolloffay @MovieStoreGuy
• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MovieStoreGuy]

Do you have an example of how you would use the headers from kafka as part of the data that is sent?

[VihasMakwana]

Yes, I would like to give you an example using Splunk Exporter:

Let's suppose I send the following event:
Event content: {"key": "value"}
Header: {"index": "main", "source": "custom"}

I eventually want this data to send in "main" index on Splunk.
I can't do this as the current implementation ignores the kafka headers and only forwards the event.
If I have the header data, I would use some processors to parse that data in the later stage of the pipeline.

This was just an example from my pov, there might be others as well.

[MovieStoreGuy]

Unfortunately the collector doesn't have a current mechanism to pass meta information from one receiver to another that can associate it with data (interestingly enough considering that this is tracing's jam).

While it is possible to extract those headers for you, it is another request to have components consume that meta information that could be provided to those components.

[VihasMakwana]

While it is possible to extract those headers for you, it is another request to have components consume that meta information that could be provided to those components.

Yeah, I agree. I can actually create a PR myself if you're open to taking a look. I'll think this through one more time.
I just want kafkareceiver to attach the header metadata and I can find a way by myself to use them.
We can mark this as optional in the configuration.

[VihasMakwana]

@MovieStoreGuy ^^

[MovieStoreGuy]

I am happy to review, but depending how much it touch will require some further conversation if we want to accept it upstream. If you have the time to create a PR as a PoC of what could happen, I would be interested to see.

Unfortunately, I don't have the time to spike this myself.

[VihasMakwana]

Yeah, no worries, I planned working on this before creating the issue, so no issues there.

If you cna assign this to me.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[VihasMakwana]

@MovieStoreGuy can you reopen this in any way? I can't see any button.
Also, can you take a look at the PR. i think it looks good

[VihasMakwana]

Completed via #24367