[Metricbeat] Add Kafka JMX metricsets

[ChrsMark]

Implements kafka.broker, kafka.consumer, kafka.producer metricset of #13366.

Depends on:

• Move light modules to OSS #14369
• Replace generate_imports python script with mage imports #14498
• Add light modules of oss as source in docs collector #14483

Manual Testing

Broker metricset

In order to prepare the proper environment to test these metricsets, the quickest way is to reuse the Docker env of the module. This can be achieved by first of all running the integration tests of the module which will take care of bringing up the container with the proper configuration.

To run the tests:
go test -v -tags=integration github.com/elastic/beats/metricbeat/module/kafka/partition

Verify that tests are successful and you can go on to manual testing. After the tests have finished the container is still alive so we can use it for the manual testing too.

Check what are the exported ports from Docker with a docker ps:

5da9631cfa25        docker.elastic.co/observability-ci/beats-integration-kafka:2.1.1-1   "/run.sh"           23 minutes ago      Up 23 minutes (healthy)   2181/tcp, 0.0.0.0:32771->8774/tcp, 0.0.0.0:32770->8775/tcp, 0.0.0.0:32769->8779/tcp, 0.0.0.0:32768->9092/tcp   metricbeat_kafka_1

In this example the mapping is:

• broker port: 0.0.0.0:32769->8779
• producer port: 0.0.0.0:32770->8775
• consumer port: 0.0.0.0:32771->8774

So we configure the metricsets' hosts accordingly.

Enable the Kafka module and use the following configuration in modules.d/kafka.yml:

# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:32769"]

# Metrics collected from a Java Kafka consumer using Jolokia
- module: kafka
  metricsets:
    - consumer
  period: 10s
  hosts: ["localhost:32771"]

# Metrics collected from a Java Kafka producer using Jolokia
- module: kafka
  metricsets:
    - producer
  period: 10s
  hosts: ["localhost:32770"]

Run Metricbeat and verify that metrics are being collected from all of the 3 metricsets we configured.

When done remove the testing container.

[jsoriano]

This is looking good, I am requesting only some small changes in Dockerfile and configuration.

[jsoriano]

curl was already installed, you can use it, and you can also use retry as is being used to download kafka.

[jsoriano]

Remove commented out code.

[jsoriano]

As the hosts in the config should be different for the different metricsets, we could put it in different blocks in the reference configuration, also adding a note about the expected host, something like what we do in the kubernetes module, something like this:

# Kafka metrics collected using the Kafka protocol
- module: kafka
  enabled: true
  metricsets:
    - consumergroup
    - partition
  period: 10s
  hosts: ["localhost:9092"]

# Metrics collected from a Kafka broker using Jolokia
#- module: kafka
#  enabled: true
#  metricsets: ['broker']
#  period: 10s
#  hosts: ['localhost:8779']

# Metrics collected from a Java Kafka consumer using Jolokia
#- module: kafka
#  enabled: true
#  metricsets: ['consumer']
#  period: 10s
#  hosts: ['localhost:8775']

# Metrics collected from a Java Kafka producer using Jolokia
#- module: kafka
#  enabled: true
#  metricsets: ['producer']
#  period: 10s
#  hosts: ['localhost:8774']

[jsoriano]

We may need a similar separation as the one suggested above for the reference config. Or we could have just one config.yml with the suggestions mentioned before and remove the config.reference.yml.

[jsoriano]

Note to myself 🙂 consider separating it in several entrypoints for #14087.
Don't do it here because it will only complicate this PR and it is not needed by now.

[jsoriano]

We may add here also a link to the Jolokia module for the compatibility notes there.

[jsoriano]

mbeans should be always in the same field at the root level (something like jmx.mbean).

This is probably something to change in the Jolokia module, so we can keep this here by now.

[jsoriano]

We may want to explicitly mention that this only works for consumers implemented in Java, and same thing in the producer metricset.

Btw:

Suggested change

	This metricset periodically fetches JMX metrics from Kafka Producers JMX.
	This metricset periodically fetches JMX metrics from Kafka Consumers JMX.

[jsoriano]

We should also mention something about the metricsets requiring Jolokia in the module documentation (metricbeat/module/kafka/_meta/docs.asciidoc).

[ChrsMark]

Merging this one since Travis succeeded and Jenkins failure is irrelevant.

Tests will be added in a follow up PR.

[ChrsMark]

Regression should be fixed now with the fixes at #15957.

[sayden]

Tested manually, good work!