Support a max bytes per second on beats protocol

[gingerwizard]

Currently Filebeat seems to push data to Logstash at the rate it can tail a file. For users where the rate of log generation is variable this can result in high bandwidth utilization - sometimes unacceptably high.

This ticket proposes introducing a max_bytes_per_second setting in the beats protocol to Logstash. If the log files being followed are written at a greater rate than this setting value, Filebeat will throttle the rate at which it tails the log file - potentially resulting in it "falling behind" . In this case periods of lower activity would be used to catch up.

Discussion required as to how this would impact other beats. It may be more appropriate for this to be a Filebeat specific setting e.g. max_read_bytes_per_second.

[phoenixy2k]

+1

[biosed]

+1 this, would be quite handy

[ruflin]

For more discussions also see https://github.com/elastic/filebeat/issues/227

[gtt116]

+1 for this, it would be nice to have a QOS.

[harpreet-sawhney]

+1

[abhishekjiitr]

+1

[alexander-marquardt]

It would be useful to limit the number of events per second that filebeat can send. This would be useful for example if an application server that is monitored by Filebeat starts to generate millions of Errors, which could otherwise overload Logstash and Elastic with a huge spike in traffic. I have seen this exact scenario cause issues while at a client.

[amomchilov]

@alexander-marquardt Yep, I had the exact same issue. If the connection was ever interuppted, a backlog built up, which was dumped as fast as FileBeat/Elasticserach could negotiate to go, which was way faster than our network infrastructure could take (over 30 Gb/s, IIRC), which overloaded our load balancers. Sad times.

[mostlyjason]

I created a new meta issue to track this. Closing this one in favor of https://github.com/elastic/beats/issues/17716

[amomchilov]

@mostlyjason "This issue has been moved to a repository you don't have access to."

[mostlyjason]

@amomchilov posted here #17775