[exporter/kafka] Replace "topic" setting by "traces_topic", "logs_topic" and "metrics_topic"

[aklemp]

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

Inspired by #32735 because it is a related problem:

When the setting "topic" is not specified, the same kafka exporter config can be used in all three pipelines if the topic names match the default values:

exporter:
  kafka:

pipelines: 
  metrics:
     exporters: [kafka] # publishes topic otlp_metrics
  logs:
     exporters: [kafka] # publishes topic otlp_logs
  traces:
     exporters: [kafka] # publishes topic otlp_spans

If the topic is set to any value, this structure will work in exporter perspective.

exporter:
  kafka:
    topic: custom_traces_topic

pipelines: 
  metrics:
     exporter: [kafka] # publishes topic custom_traces_topic
  logs:
     exporter: [kafka] # publishes topic custom_traces_topic
  traces:
     exporter: [kafka] # publishes topic custom_traces_topic

What happens in this case is that the three exporters will send to the same topic. This is a race condition that will succeed in 1/3 of scenarios at the receiving end (see #32735).

To avoid this problem, the user must create three different exporters for each pipeline to set custom topic names. This is error prone and inconsistent with the default behavior that allows having one exporter for all three pipelines with the default topic names.

Describe the solution you'd like

Having three different topic names by default but being able to override it with only a single one is a strange feature.
Just create three topic properties of the kafka exporter:

exporter:
  kafka:
    traces_topic: custom_traces_topic # default otlp_spans
    metrics_topic: custom_metrics_topic # default otlp_metrics
    logs_topic: custom_logs_topic # default otlp_logs

pipelines: 
  metrics:
     exporters: [kafka] # publishes topic custom_metrics_topic
  logs:
     exporters: [kafka] # publishes topic custom_logs_topic
  traces:
     exporters: [kafka] # publishes topic custom_traces_topic

Alternative definition (solution should match for exporter and receiver):

exporter:
  kafka:
    topic:
      traces: custom_traces_topic # default otlp_spans
      metrics: custom_metrics_topic # default otlp_metrics
      logs: custom_logs_topic # default otlp_logs

Describe alternatives you've considered

The documentation gives some hints about determining the actual topic:

1. The client application sending telemetry data to OpenTelemetry should not be concerned with setting topic names in attributes that are used internally to transport OpenTelemetry information using Kafka.
2. The context could be configured with topic names.
   • found no example how to configure that
   • need to have some logic to determine from telemetry data to configure the topic to use
   • evaluated for every message
   • more complex setup than simply defining three static properties
3. Feature enhancement of this ticket.

Current workaround: define three exporters with all properties redundant except the topic property and use them individually in three pipelines.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.