[Metricbeat] Add New Metrics to the Kafka Module

[sorantis]

Currently the Kafka module in Metricbeat supports two Kafka entities - consumergroup, partition. Customer is requesting to expand the modules with metricsets covering the following metrics:

- kafka.net.bytes_out
- kafka.net.bytes_in
- kafka.net.bytes_rejected
- kafka.messages_in
- kafka.request.channel.queue.size
- kafka.request.fetch.failed
- kafka.request.fetch.failed_per_second
- kafka.request.produce.failed_per_second
- kafka.request.produce.failed
- kafka.replication.leader_elections
- kafka.replication.unclean_leader_elections
- kafka.session.zookeeper.disconnect
- kafka.session.zookeeper.expire
- kafka.session.zookeeper.readonly
- kafka.session.zookeeper.sync
- kafka.log.flush_rate
- kafka.consumer.delayed_requests
- kafka.consumer.expires_per_second
- kafka.producer.available_buffer_bytes
- kafka.producer.batch_size_avg
- kafka.producer.delayed_requests
- kafka.producer.expires_per_seconds
- kafka.producer.batch_size_max
- kafka.producer.record_send_rate
- kafka.producer.record_retry_rate
- kafka.producer.record_error_rate
- kafka.producer.records_per_request
- kafka.producer.record_size_avg
- kafka.producer.record_size_max
- kafka.producer.request_rate
- kafka.producer.response_rate
- kafka.producer.bytes_out
- kafka.producer.message_rate
- kafka.producer.io_wait
- kafka.consumer.max_lag
- kafka.consumer.fetch_rate
- kafka.consumer.bytes_consumed
- kafka.consumer.bytes_in
- kafka.consumer.messages_in
- kafka.consumer.zookeeper_commits
- kafka.consumer.kafka_commits
- kafka.consumer.records_consumed
- kafka.topic.messages_in
- kafka.topic.net.bytes_out
- kafka.topic.net.bytes_in
- kafka.topic.net.bytes_rejected

More information about these metrics can be found here.

[jsoriano]

As they are JMX metrics I guess that this would have to be implemented as a light module based on the Jolokia one. We might want to wait to have this before #13316.

[sorantis]

+1, except the kafka.session.zookeeper.* metrics, which are mentioned here.

[ChrsMark]

This will be implemented in 3 different metricsets:

• kafka.broker : [Metricbeat] Add Kafka JMX metricsets #14330
• kafka.producer : [Metricbeat] Add Kafka JMX metricsets #14330
• kafka.consumer : [Metricbeat] Add Kafka JMX metricsets #14330

[ChrsMark]

Final mapping:

- kafka.net.bytes_out -> kafka.broker.net.out.bytes_per_sec
- kafka.net.bytes_in -> kafka.broker.net.in.bytes_per_sec
- kafka.net.bytes_rejected -> kafka.broker.net.rejected.bytes_per_sec
- kafka.messages_in -> kafka.broker.messages_in
- kafka.request.channel.queue.size  -> kafka.broker.request.channel.queue.size
- kafka.request.fetch.failed_per_second -> kafka.broker.request.fetch.failed_per_second
- kafka.request.produce.failed_per_second -> kafka.broker.request.produce.failed_per_second
- kafka.request.produce.failed -> kafka.broker.request.produce.failed_per_second
- kafka.request.fetch.failed -> kafka.broker.request.fetch.failed_per_second
- kafka.replication.leader_elections -> kafka.broker.replication.leader_elections
- kafka.replication.unclean_leader_elections -> kafka.broker.replication.unclean_leader_elections
- kafka.session.zookeeper.disconnect -> kafka.broker.session.zookeeper.disconnect
- kafka.session.zookeeper.expire -> kafka.broker.session.zookeeper.expire
- kafka.session.zookeeper.readonly -> kafka.broker.session.zookeeper.readonly
- kafka.session.zookeeper.sync -> kafka.broker.session.zookeeper.sync
- kafka.log.flush_rate -> kafka.broker.log.flush_rate
- kafka.producer.available_buffer_bytes -> kafka.producer.available_buffer_bytes
- kafka.producer.batch_size_avg -> kafka.producer.batch_size_avg
- kafka.producer.batch_size_max -> kafka.producer.batch_size_max
- kafka.producer.record_send_rate -> kafka.producer.record_send_rate
- kafka.producer.record_retry_rate -> kafka.producer.record_retry_rate
- kafka.producer.record_error_rate -> kafka.producer.record_error_rate
- kafka.producer.records_per_request -> kafka.producer.records_per_request
- kafka.producer.record_size_avg -> kafka.producer.record_size_avg
- kafka.producer.record_size_max -> kafka.producer.record_size_max
- kafka.producer.request_rate -> kafka.producer.request_rate
- kafka.producer.response_rate -> kafka.producer.response_rate
- kafka.producer.bytes_out -> kafka.producer.out.bytes_per_sec
- kafka.producer.io_wait -> kafka.producer.io_wait
- kafka.consumer.max_lag -> kafka.consumer.max_lag
- kafka.consumer.fetch_rate -> kafka.consumer.fetch_rate
- kafka.consumer.bytes_consumed -> kafka.consumer.bytes_consumed
- kafka.consumer.bytes_in -> kafka.consumer.in.bytes_per_sec
- kafka.consumer.records_consumed -> kafka.consumer.records_consumed
- kafka.topic.messages_in -> kafka.broker.topic.messages_in
- kafka.topic.net.bytes_out -> kafka.broker.topic.net.out.bytes_per_sec
- kafka.topic.net.bytes_in -> kafka.broker.topic.net.in.bytes_per_sec
- kafka.topic.net.bytes_rejected -> kafka.broker.topic.net.rejected.bytes_per_sec
- kafka.consumer.zookeeper_commits -> kafka.consumer.zookeeper_commits
- kafka.consumer.kafka_commits -> kafka.consumer.kafka_commits
- kafka.producer.message_rate -> kafka.producer.message_rate
- kafka.consumer.messages_in -> kafka.consumer.messages_in


Not found for current versions:
- kafka.producer.delayed_requests -> Not found
- kafka.producer.expires_per_seconds -> Not found

[ChrsMark]

Last PR adding tests and final missing fields is #14677.

@sorantis feel free to comment on the final list of fields or give a green light for closing this after the PR is merged 🙂.

[sorantis]

LGTM