Cherry-pick #14330 to 7.x: [Metricbeat] Add Kafka JMX metricsets

[ChrsMark]

Cherry-pick of PR #14330 to 7.x branch. Original message:

Implements kafka.broker, kafka.consumer, kafka.producer metricset of #13366.

Depends on:

• Move light modules to OSS #14369
• Replace generate_imports python script with mage imports #14498
• Add light modules of oss as source in docs collector #14483

Manual Testing

Broker metricset

In order to prepare the proper environment to test these metricsets, the quickest way is to reuse the Docker env of the module. This can be achieved by first of all running the integration tests of the module which will take care of bringing up the container with the proper configuration.

To run the tests:
go test -v -tags=integration github.com/elastic/beats/metricbeat/module/kafka/partition

Verify that tests are successful and you can go on to manual testing. After the tests have finished the container is still alive so we can use it for the manual testing too.

Check what are the exported ports from Docker with a docker ps:

5da9631cfa25        docker.elastic.co/observability-ci/beats-integration-kafka:2.1.1-1   "/run.sh"           23 minutes ago      Up 23 minutes (healthy)   2181/tcp, 0.0.0.0:32771->8774/tcp, 0.0.0.0:32770->8775/tcp, 0.0.0.0:32769->8779/tcp, 0.0.0.0:32768->9092/tcp   metricbeat_kafka_1

In this example the mapping is:

• broker port: 0.0.0.0:32769->8779
• producer port: 0.0.0.0:32770->8775
• consumer port: 0.0.0.0:32771->8774

So we configure the metricsets' hosts accordingly.

Enable the Kafka module and use the following configuration in modules.d/kafka.yml:

# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:32769"]

# Metrics collected from a Java Kafka consumer using Jolokia
- module: kafka
  metricsets:
    - consumer
  period: 10s
  hosts: ["localhost:32771"]

# Metrics collected from a Java Kafka producer using Jolokia
- module: kafka
  metricsets:
    - producer
  period: 10s
  hosts: ["localhost:32770"]

Run Metricbeat and verify that metrics are being collected from all of the 3 metricsets we configured.

When done remove the testing container.

[jsoriano]

Not related? CI jobs seem to be failing because of this. Run mage fmt update and commit the changes.

[ChrsMark]

Yes the whole file was removed on #14553 accidentally. Will run the updates, thank you!