core: minimal synchronous scheduler master

[matthewfala]

Minimal Synchronous Scheduler - Master

Please leave comments on the 1.9 PR #6339

Summary

Implement a synchronous task scheduler plugin option allowing for the cloudwatch_logs plugin to opt into to allow for migrate to the Async Network stack.

Issue

Due to limitations of the CloudWatch API in processing PutLogEvents network requests synchronously, Fluent Bit sends data to CloudWatch using a less supported “synchronous” networking stack. This stack is prone to indefinite hangs and segfaults and only works well when used with a fine tuned configuration.
See: #6140 and #6329

Investigative Efforts

While at first we tried to resolve the networking hang issues found in the synchronous network stack by adding OpenSSL error handling, DNS Timeouts, and enabling unidirectional TLS shutdowns, these efforts only made fluent bit go from failing once in 5 minutes without the changes to once in 5 hours with the changes - when tested under a high load failure case. We determined that it would take too much effort to isolate synchronous network stack issues and decided to invest efforts switching to the widely used Fluent Bit asynchronous network stack.

• Open SSL error handling (https://github.com/matthewfala/fluent-bit/commits/instrument-1.9.8-net-layer)
• Remove bidirectional tcp shutdown (https://github.com/matthewfala/fluent-bit/commits/instrument-1.9.8-net-layer)
• Switch to async DNS (https://github.com/matthewfala/fluent-bit/commits/instrument-1.9.8-net-layer)

Solution

Our proposed solution is to migrate the Cloudwatch Logs output plugin to Fluent Bit’s asynchronous network stack.

Switching to Async Network Stack

CloudWatch API Synchronous Usage Requirements
CloudWatch relies on the Synchronous networking to ensure that CloudWatch Logs PutLogEvents requests are done sequentially.

Normally when the asynchronous network stack is used, Fluent Bit context switches in the next batch of logs into processing when the previous batch yields on a network call. This defeats the desired sequential PutLogEvents execution required by CloudWatch.

Existing Core Synchronous Scheduler
In order to enforce sequential processing of log data when the asynchronous network stack is used, we opt our CloudWatch Logs plugin into a Fluent Bit Core synchronous task scheduler which limits one batch of logs to be processed at a time, essentially using the asynchronous networking stack in a synchronous manner.

A bottleneck was discovered in the Fluent Bit Core Synchronous scheduler, which limits processing logs to 1 batch per second (or per flush interval).

New Performant Core Synchronous Scheduler
A performant new core scheduler was written by the FireLens team that removes this 1 batch per second restriction while keeping the one batch at a time processing restriction in place. The CloudWatch Plugin opts into the performant Synchronous Scheduler implementation and uses the asynchronous network stack.

For plugins that opt into FLB_OUTPUT_SYNCHRONOUS by setting that as a plugin flag, there will be a limit of 1 task per output_instance worker group.

Testing and Results

Unit Testing

A series of 24 hour tests were conducted on Fluent Bit 1.9x with the patch with and without Valgrind. No network hangs were observed on 1.9 and no memory leaks were introduced by the patch.

A 24 hour test was conducted on Fluent Bit 2.0x with the patch. No network hangs were observed.

Parallel Long Running Durability Tests

To simulate the customer’s long running execution of Fluent Bit, 40-100 ECS FireLens test tasks per test were run in parallel to accumulate cumulative running time and gain confidence in the patch.

The following is a stability matrix outlining the patches impact on Fluent Bit’s durability rating which is described lowerbounded average hours to failure (HTF)

Fluent 1.9x (AWS For Fluent Bit Official Release)

_	Patch	No Patch
Keepalive On	[2] Very Stable (+3500h)	[1,8] Segfault on some network errors after throttling limits (~80h)
Keepalive Off	[3] Somewhat stable (~2000h)	Cloudwatch Hang(~0.08h)

Fluent 2.0x (Upstream Fluent Bit Distribution)

_	Patch	No Patch
Keepalive On	[7] Segfault on connection time out(~100h)	[4] Stable(+2200h)
Keepalive Off	[6] Segfault on connection timeout (~100h)	Cloudwatch Hang(~0.08h)

Segfaults were infrequently observed with the patch on 2.0 and may be attributed to 2.0x instabilities.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.