fluentd_output_status_buffer_queue_length not showing accurate count

[buch11]

Hey Guys,

I have fluentd setup which publishes metrics to prometheus server. I have an HTTP_OUT plugin enabled as my http output. What I am after is the count of messages which are not flushed to HTTP_OUT yet.

Test case :

• Bring down the server to which HTTP_OUT plugin is talking to force fluentd to buffer all the messages.
• Send around 100 messages to fluentd.
• Observe metrics

What I am observing is fluentd_output_status_emit_records and fluentd_input_status_num_records_total have the same value as the number of records that I had sent to fluentd. However, fluentd_output_status_buffer_queue_length is showing only 5 as a constant count.

Can you guide me as to why this is the case? or which metrics should I use to monitor the messages that are yet to be flushed?

Fluentd -> dockerhub.artifactory.ai.cba/fluent/fluentd:v1.10.4-1.0
Prometheus plugin -> fluent-plugin-prometheus -v 1.6.1