[exporter/kafka] Logs with custom schema

[nanorobocop]

Is your feature request related to a problem? Please describe.
We want to send logs to Kafka using our custom schema.
Kafka exporter sends logs in defined LogsData format.
In our case we already have predefined schema for logs in our storage, so we want to send logs using our own schema.
Currently Kafka exporter does not allows to modify schema.

Describe the solution you'd like
I'm thinking about several different approaches how to implement custom schema for kafka exporter:

• Send only Attributes.
  • Attributes are dynamic map[string]Any fields, so would be easy to use.
• Mapping Fields.
  • This is similar to Elasticsearch Fields. Those Fields will be added as top level fields of original record.

First approach is more generic I think.
It requires only attributes modification in standard opentelemetry-collector pipeline.

I'd like to hear what community thinks about it or if any other ideas.

Describe alternatives you've considered

Additional context

[github-actions]

Pinging code owners: @pavolloffay @MovieStoreGuy

[nanorobocop]

This is probably depends on #12623
Custom schema should be applied to individual records, not set of records.

[MovieStoreGuy]

I think this warrants a further discussion than just applying to the KafkaExporter since this could also apply to the Kinesis Exporter, FileExporter and I am sure more in future.

I need to think more about this because it also comes with a performance impact as well

[nanorobocop]

Thanks for comment!

My implementation consists of 2 parts:

• [exporter/kafka] Support sending individual log records [exporter/kafka] Support sending individual log records #12657
• [exporter/kafka] Send log record attributes marshaler (for custom schema) [exporter/kafka] Send log record attributes marshaler (for custom schema) #12658

Custom schema implemented just by sending Attributes (which is dynamic map, could be constructed during Processing phase). So similar approach could be done in other exporters as well.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.