Fluentbit tail missing some big-ish log line even with Buffer_Max_Size set to high value

[kiich]

Bug Report

Describe the bug
One of our app is logging to stdout and fluentbit is able to capture the stdout log from (docker) log path successfully and send onto Splunk which we can see fine.

However, we have noticed 1 type of log line is always missing from splunk, even though we can see if in the pod log.

The log line in question is a big-ish 1 line which made me look at the tuning of:

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Docker_Mode      On
    Docker_Mode_Flush 4
    Refresh_Interval 5
    Mem_Buf_Limit    32MB
    Buffer_Max_Size  2048KB
    Skip_Long_Lines  On

And the log line is question, has this stats:

wc -l long-line.log
       1 long-line.log

wc -c long-line.log
   20904 long-line.log

du -sk long-line.log
24	long-line.log

so not particulary big in size so i assumed the Skip_Long_Lines On with Buffer_Max_Size set to 2048KB was fine.

I can't check if fluentbit is not capturing it or not sending it but would be awesome if i can find this out as i am pretty sure Splunk can handle this log line fine.

fluentbit log shows nothing but i guess thats because its not in debug which i can set to.

[kiich]

Also to add - once this problematic log line appears in the log, we don't see any subsequent log lines neither in splunk - which seems to match the behaviour of not setting Skip_Long_Lines to true - but we are in this case?

[kiich]

Using Fluent Bit v1.2.2

[kiich]

Linking related issues as it now seems the issue is not the big log line but the fact the log file in question is both:

1. symlink to another file
2. That file gets rotated

And when that log file gets filled up super quickly, it seems to then cause this issue of lost log in splunk.

#1712
#1108
#375
#1118

[yusufgungor]

We got the same problem, is there any update?

[hasakilol]

Maybe you need to comment this out:
Docker_Mode On

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.