Let's create a Global equivalent of mem_buf_limit or some new memory limiter

[PettitWesley]

One of the most common questions I get from users is how to prevent OOMKills and limit the memory of Fluent Bit.

Today we have the mem_buf_limit setting on the input definitions that can constrain the memory that each input can use to ingest data. This is a powerful setting, but I think for novice/beginner/less sophisticated users, we can do better.

I have two ideas that I want to put forward.

[Preferred] Proposal 1: Global equivalent of mem_buf_limit

The simplest solution is a global/service level version of the Mem_Buf_Limit setting which applies to all inputs. I have an email thread with Eduardo where he says he is open to this idea.

The setting might look like:

[SERVICE]
    Total_Input_Memory  100M

And this would limit all inputs to not use more than 100M of memory for their buffers, similar to how Mem_Buf_Limit does this for a single input. If the user has both this global setting and a mem_buf_limit on an input, then whichever limit is hit first will come into affect.

Proposal 2: A real memory allocation limiter

Another idea that I had was that since all Fluent Bit code should either use flb_malloc or flb_calloc to allocate almost any memory used (with the exception of libraries like msgpack that we use that allocate their own memory) or may be via some magic with a custom implementation of the underlying calloc and malloc functions, we could actually build a true memory limiting feature that wouldn't allow plugins to allocate more memory if the total allocation was greater than some value. Since we always have checks after every allocation call, this should be safe and would stop FB from functioning but won't crash it.

This probably is a bad idea since Fluent Bit would need new memory to flush backlog data and so it could get into a state where no plugin can free any memory since that requires doing work which requires allocating new memory...

[PettitWesley]

Got a request for this from a user on the AWS repo: aws/aws-for-fluent-bit#378

[matthewfala]

Task Memory Limits

Fluent Bit currently has two main memory users:

• Buffer
• Tasks

3 if you count engine, but engine/worker threads have very limited memory consumption.

So far we only have buffer memory limits in fluent bit.

This is a problem because users don't just care about the buffer memory usage but care also about the task memory usage.

We have a way to limit buffer memory usage, but there is currently no way to limit task memory usage.
I propose that a new configuration option for input and output plugins is added called task_memory_limit (or similar).

Then fluent bit can hold each task accountable to it's memory limit setting. It can deny a task of additional memory grants for a task it's quota is exceeded and mark the task for deletion / let the task fail.

Also, for a Fluent bit total_memory limit setting, it can reserve memory for starting tasks. A task should only be started if there is enough memory to meet the task_memory_limit setting.

[r0b0ji]

I had also created an issue #5685 but it got closed because of no activity and staleness.