[WIP] Throttling filter

[onorua]

There are cases when you need to handle only the load you agreed on, and drop the rest (some sort of internal service with SLA). This plugin does exactly this. We have some rate per second, and we drop messages above it. It implements leaky bucket, based on sliding window algorithm.

There are couple of things we could improve on though (looking forward to your suggestions):

1. I would like to have a background time ticker, don't know how to make it using fluent-bit framework. Should I wake up on the event, or some background thread is enough.
2. Do you think we need different time ranges, like per minute, per hour etc?
3. Do think anything can be improved? I got 1 core nocked out for more than 100k req/sec.

What do you think about the idea in general?

[onorua]

@edsiper should I create something similar to flb_input_set_collector_time for filter? Do you think such a functionality can be merged when done, or this is something you don't want in mainline?

[edsiper]

hi @onorua

If this plugin is for a production use-case I am happy to have it there.

regarding your questions:

1. I would like to have a background time ticker, don't know how to make it using fluent-bit framework
   Ideally this need to have a time_collector into the filters which don't exists at the moment. I think it's necessary, I can add the bits for that.

2. Should I wake up on the event, or some background thread is enough.
   I think Fix include paths #1 is enough

3. Do you think we need different time ranges, like per minute, per hour etc?
   do you have more use-cases ?, I would like to learn more from that.

4. Do think anything can be improved? I got 1 core nocked out for more than 100k req/sec.
   do you mean to increase performance ?

[edsiper]

@onorua are you still interested into merge improve this feature ?

[onorua]

Yes, we have this in production for some time and really happy about it. Give me couple of days for tests coverage. I'll split this PR into 2 PRs one for throttling and another one for elastic search index regexp.

[onorua]

closing in favor of #511