Skip to content

Pipeline: filters: throttle: style: #1701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 30, 2025
+21 −23
Merged

Pipeline: filters: throttle: style: #1701

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e8733df
Pipeline: filters: throttle: style:
esmerel May 29, 2025
d39ce42
Apply suggestions from code review
esmerel May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 21 additions & 23 deletions pipeline/filters/throttle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Throttle

The _Throttle Filter_ plugin sets the average _Rate_ of messages per _Interval_, based on leaky bucket and sliding window algorithm. In case of overflood, it will leak within certain rate.
The _Throttle_ filter sets the average `Rate` of messages per `Interval`, based on the leaky bucket and sliding window algorithm. In case of flooding, it will leak at a certain rate.

## Configuration Parameters
## Configuration parameters

The plugin supports the following configuration parameters:

| Key | Value Format | Description |
| :--- | :--- | :--- |
| Rate | Integer | Amount of messages for the time. |
| Window | Integer | Amount of intervals to calculate average over. Default 5. |
| Interval | String | Time interval, expressed in "sleep" format. e.g 3s, 1.5m, 0.5h etc |
| Print\_Status | Bool | Whether to print status messages with current rate and the limits to information logs |
| `Rate` | `Integer` | Amount of messages for the time. |
| `Window` | `Integer` | Amount of intervals to calculate average over. Default: `5`. |
| `Interval` | `String` | Time interval, expressed in `sleep` format. For example, `3s`, `1.5m`, `0.5h`. |
| `Print_Status` | `Bool` | Whether to print status messages with current rate and the limits to information logs. |

## Functional description

Expand All @@ -23,9 +23,7 @@ Window 5
Interval 1s
```

You would received 1 message in the first second, 3 messages second, and 5 third.
Disregard that Window is actually 5, because the configuration uses `slow` start
to prevent flooding during the startup.
You would receive 1 message in the first second, 3 messages second, and 5 third. Disregard that Window is actually 5, because the configuration uses `slow` start to prevent flooding during the startup.

```text
+-------+-+-+-+
Expand All @@ -35,7 +33,7 @@ to prevent flooding during the startup.
+-----+
```

But as soon as we reached Window size \* Interval, we will have true sliding window with aggregation over complete window.
But as soon as you reach `Window size * Interval`, you will have true sliding window with aggregation over complete window.

```text
+-------------+
Expand All @@ -45,7 +43,7 @@ But as soon as we reached Window size \* Interval, we will have true sliding win
----------+
```

When we have average over window is more than Rate, we will start dropping messages, so that
When the average over window is more than `Rate`, Fluent Bit starts dropping messages, so the following:

```text
+-------------+
Expand All @@ -65,11 +63,11 @@ will become:
+---------+
```

As you can see, last pane of the window was overwritten and 1 message was dropped.
The last pane of the window was overwritten and 1 message was dropped.

### Interval vs Window size
### Interval versus Window size

You might noticed possibility to configure _Interval_ of the _Window_ shift. It is counter intuitive, but there is a difference between two examples above:
You might notice it's possible to configure the `Interval` of the `Window` shift. It's counter intuitive, but there is a difference between the two previous examples:

```text
Rate 60
Expand All @@ -85,7 +83,7 @@ Window 300
Interval 1s
```

Even though both examples will allow maximum Rate of 60 messages per minute, first example may get all 60 messages within first second, and will drop all the rest for the entire minute:
Even though both examples will allow maximum `Rate` of 60 messages per minute, the first example might get all 60 messages within first second, and will drop all the rest for the entire minute:

```text
XX XX XX
Expand All @@ -97,24 +95,24 @@ XX XX XX
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```

While the second example will not allow more than 1 message per second every second, making output rate more smooth:
While the second example won't allow more than 1 message per second every second, making output rate more smooth:

```text
X X X X X X
XXXX XXXX XXXX XXXX XXXX XXXX
+-+-+-+-+-+--+-+-+-+-+-+-+-+-+-+
```

It may drop some data if the rate is ragged. I would recommend to use bigger interval and rate for streams of rare but important events, while keep _Window_ bigger and _Interval_ small for constantly intensive inputs.
Fluent Bit might drop some data if the rate is ragged. Use bigger intervals and rates for streams of rare but important events, while keeping `Window` bigger and `Interval` smaller for constantly intensive inputs.

### Command Line
### Command line

> Note: It's suggested to use a configuration file.
It's suggested to use a configuration file.

The following command will load the _tail_ plugin and read the content of _lines.txt_ file. Then the _throttle_ filter will apply a rate limit and only _pass_ the records which are read below the certain _rate_:
The following command will load the Tail plugin and read the content of the `lines.txt` file. Then, the Throttle filter will apply a rate limit and only pass the records which are read below the `rate`:

```text
$ bin/fluent-bit -i tail -p 'path=lines.txt' -F throttle -p 'rate=1' -m '*' -o stdout
```shell
bin/fluent-bit -i tail -p 'path=lines.txt' -F throttle -p 'rate=1' -m '*' -o stdout
```

### Configuration File
Expand All @@ -136,4 +134,4 @@ $ bin/fluent-bit -i tail -p 'path=lines.txt' -F throttle -p 'rate=1' -m '*' -o s
Match *
```

The example above will pass 1000 messages per second in average over 300 seconds.
This example will pass 1000 messages per second in average over 300 seconds.