Truncate long lines more than buffer_max_size for Tail Input Plugin #9990
Description
Is your feature request related to a problem? Please describe.
Currently the Tail Input plugin doesn't process log messages that are more than the buffer_max_size and just skips them in case skip_long_lines is enabled.
Describe the solution you'd like
There should be an option to truncate the message (maybe a new parameter max_log_size) to either default to buffer_max_size or be configurable to a given size.
Currently the CloudWatch Logs Output plugin has this truncating feature
The AWS CloudWatch Agent does the same logic of truncating big log lines to 256KB
After truncation it reopens the file for tailing -
Describe alternatives you've considered
Additional context
I am trying to use a Tail Input Plugin with a HTTP Output plugin where the input log line can be arbitrary long but the HTTP Output can only process 256KB of a single log line.
I would want FluentBit Tail plugin to be able to truncate the big log line >256KB. Also for my use case the mem_buf_limit can not be more than 1MB thus we can not fit very large log lines into the buffer and use Filters to truncate the log lines.