Fluentd ignores the buffer queue_length_limit configuration value

[Ponyboy47]

fluentd 0.14.9
CentOS 7.3
Ruby 2.3

Config:

<source>
  @type tail
  path /opt/search/appliance5/logs/ps-usage.*-*.log, /opt/search/compoundquery/logs/ps-cq-usage-*
  pos_file /opt/search/appliance5/logs/ps-usage.log.pos
  format /(?<logEntry>{.*)/
  tag logentry
  read_from_head true
</source>

<filter logentry>
  @type psxml
</filter>

<match logentry>
  @type http_buffered
  <buffer>
    type file
    flush_interval 30s
    path /var/log/fluentd/buffer.*
    chunk_limit_size 15m
    queue_length_limit 34134
  </buffer>
  endpoint_url http://localhost:30000/cache/addrecords?store=%Y
  http_retry_statuses 500, 403, 404, 400
  http_read_timeout 30
  http_open_timeout 30
  http_keep_alive_timeout 600
  record_contains <ps:record, <log_entry>, </log_entry>, </ps:record>
</match>

Recently ran into an interesting issue with fluentd.

So something strange happened where our SELinux policies got messed up on a customer's machine and fluentd was denied access whenever it tried to read from or write to buffer files. As a result, fluentd infinitely tried to create new buffer files to read from and write to. This occurred on a customer machine that is not frequently used and so we did not notice the problem for over a week, at which point fluentd had created almost 5 million completely empty buffer files. Clearly this is a massive violation of the configuration option. While the full buffer size limit was not reached (500Gb), fluentd should still stop creating buffer files once the number of files created is equal to the queue_length_limit. Especially since fluentd tries opening ALL of the buffer files at once, if I had let fluentd try and open all 5 million buffer files then the system would not have had any left for other tasks.

I'm sorry I don't have the log file right now to show you the exact error message, but it was a permissions error with opening the "current" buffer file for read/writing, then fluentd would try to create a new buffer file and try again with the new buffer file. This repeated until we noticed the problem, fixed SELinux, and deleted all the empty buffer files and then restarted fluentd.

[repeatedly]

Does this happen with latest fluentd v0.14?
We recently fixed one bug of buf_file: #1468
This is not for this case directly, but buf_file raises BufferOverflowError when can't create new files.

[nraddy]

Error "queue_length_limit" ignore happens even with latest fluent v0.14.20. It happens

[warn]: parameter 'queue_length_limit' in
@type "file"
path "/var/log/td-agent/buffer/fluentd_log_buffer"
flush_interval 1
queue_length_limit 10000
chunk_limit_size 1M
chunk_records_limit 200
flush_mode interval
flush_thread_interval 0.05
flush_thread_burst_interval 0.05
total_limit_size 1073741824
is not used.

[joker1007]

I already commented at fluent-plugins-nursery/fluent-plugin-bigquery#142.
I notice here.

queue_length_limit is invalid name.
Right config is queue_limit_length.

[repeatedly]

Yes, joker1007 is correct: https://docs.fluentd.org/v0.14/articles/buffer-plugin-overview#configuration-parameters

[Ponyboy47]

Shouldn't fluentd not even start if the config is wrong then? In which version did this name change? All the documentation I used to build our config used the name 'queue_length_limit' and since fluentd started I assumed that was the correct name. Normally config errors mean fluentd won't even start.

[kenhys]

It seems that it was resolved by #1545 (comment)