Add kafka support

[marctc]

Add support to track process (consume/fetch) and publish client operations for Kafka in non-Go services.
The PR includes an integration test with a Python example.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 78.27715% with 58 lines in your changes missing coverage. Please review.

Project coverage is 78.01%. Comparing base (84a37be) to head (d4c9563).
Report is 156 commits behind head on main.

Files	Patch %	Lines
pkg/internal/ebpf/common/kafka_detect_transform.go	77.95%	18 Missing and 10 partials ⚠️
pkg/internal/export/prom/prom.go	70.58%	10 Missing ⚠️
pkg/internal/request/span_getters.go	50.00%	8 Missing and 2 partials ⚠️
pkg/internal/export/otel/traces.go	60.00%	6 Missing ⚠️
pkg/internal/export/otel/metrics.go	80.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #890      +/-   ##
==========================================
+ Coverage   72.28%   78.01%   +5.73%     
==========================================
  Files         124      125       +1     
  Lines        8857     9124     +267     
==========================================
+ Hits         6402     7118     +716     
+ Misses       2033     1542     -491     
- Partials      422      464      +42     

Flag	Coverage Δ
integration-test	53.95% <30.33%> (-0.66%)	⬇️
k8s-integration-test	58.76% <18.72%> (?)
oats-test	36.93% <65.54%> (+1.01%)	⬆️
unittests	43.74% <64.41%> (+0.62%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[grcevski]

I fixed up bunch of things:

1. The payload we want to parse can be in the response from Kafka when the message is pushed, rather than polled.
2. I think the library we were using in the Python example has issues with Python 3.12 and I also couldn't figure out how to retry connectivity. Kafka takes some time to properly boot and the producer needs to keep trying. I changed to another library and modified the test code to use python-kafka, it's much simpler and explicit.
3. I updated the semconv to match latest https://opentelemetry.io/docs/specs/semconv/messaging/kafka/ and https://opentelemetry.io/docs/specs/semconv/messaging/messaging-metrics/.

One thing that doesn't work is the metric attributes. I can't see why we aren't adding MessagingSystem and MessagingDestinationName. Something that needs to be debugged. The metrics are split by "publish" and "processing". Processing is the receive part, but according to the spec it's more broad and should include the push of messages too.

Here's an example of my metrics:

messaging_publish_duration_count{instance="468df55efd81-3103683", job="integration-test/python3.11", service_name="python3.11"}

there should be messaging_system and messaging_destination_name attribute there.

[marctc]

Is this doing anything? Do we still need a web server in this example?

[grcevski]

Yeah it does nothing at the moment, I wasn't sure we can make a test without it. I don't mind it being there.

[marctc]

One thing that doesn't work is the metric attributes. I can't see why we aren't adding MessagingSystem and MessagingDestinationName. Something that needs to be debugged. The metrics are split by "publish" and "processing". Processing is the receive part, but according to the spec it's more broad and should include the push of messages too.

Here's an example of my metrics:

messaging_publish_duration_count{instance="468df55efd81-3103683", job="integration-test/python3.11", service_name="python3.11"}

there should be messaging_system and messaging_destination_name attribute there.

done

[grcevski]

LGTM! Awesome stuff Marc!