Is there any way to gather producer metrics?

[neoeahit]

I dont see stats_cb being exposed. Are there any plans/works to expose stats_cb so that we can consume metrics? Also possible to see a sample(types of metrics) of these metrics exposed?

[edenhill]

@hqin Can you provide a stats sample for our friend?

[mtrienis]

Do producer metrics exist yet? If so, in what version?

I'd like to latency from when the message is queued to when it's sent. As well as the number of successful (and failed) messages.

[edenhill]

@mtrienis Register a stats callback, set statistics.interval.ms, and have a look here for a description of the format

[mtrienis]

Thanks @edenhill, do you mind clarifying what version the callback function is supported on?

This is the error I'm getting using v0.92 confluent-kafka-python and v0.91 librdkafka.

----------------------------------------------------------------------
Traceback (most recent call last):
    self._producer = confluent_kafka.Producer(producer_config)
KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Property "stats_cb" must be set through dedicated .._set_..() function"}

Code

        def my_stats_callback(stats_json_str):
            import pdb; pdb.set_trace()

        producer_config = {'bootstrap.servers': ','.join(self._config['brokers']),
                           'queue.buffering.max.ms': 1,
                           'socket.blocking.max.ms': 1,
                           'statistics.interval.ms': 250,
                           'socket.timeout.ms': 300,
                           'metadata.request.timeout.ms': 250,
                           'message.send.max.retries': 2,
                           'stats_cb': my_stats_callback,
                           'default.topic.config': {'message.timeout.ms': 350,
                                                    'request.timeout.ms': 250}}

[edenhill]

You need to upgrade to confluent-kafka-python v0.9.4