Fluentbit stops sending logs to cloudwatch after some time.

[Sangeethavisa]

Describe the question/issue

I'm facing issue with forwarding my application logs to cloudwatch. It does send logs but after some time the connection gets lost between container and cloudwatch. And when I exec the fluentbit.config file again, remaining logs gets flushed to the Cloudwatch. I want all the logs to be read and pushed to the cloudwatch stream without any delay or manual execution.

Configuration

[fluentbit.txt] (https://github.com/aws/aws-for-fluent-bit/files/10015640/fluentbit.txt)

Fluent Bit Log Output

fluentbit_debug_log.txt

Cluster Details

We are having an ECS cluster in which fluentbit container runs as a separate service with a container. This container has been mounted with the EFS ( which has all the applications logs stored in it )

Steps to reproduce issue

Build a custom fluent-bit image and launch a container which has EFS ( contains all app logs ) mounted in a path (say /log )
Dockerfile.txt

Observe in cloudwatch stream for latest logs.

Related Issues

#135

@PettitWesley Please suggest on this, as this is affecting our prod environment.

Thanks!!

[zwj102030]

Hi could you confirm on which type plugin you are using ? If you are using C plugins we suggest you switch to go-plugin at this moment. Here is detail

[PettitWesley]

Please generally check our debugging guide: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#how-do-i-tell-if-fluent-bit-is-losing-logs

And as already noted, if you are seeing that Fluent Bit just freezes sending to CW without a clear error message/explanation, then try migrating to the Golang cloudwatch plugin. See the comment above.

[PettitWesley]

Please see the note about the cloudwatch hang issue here: #542

[bansalrajat]

This issue still exists in version 2.31.11 using this helm chart
https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit/0.1.30

[mattpopa]

aws-fluent-bit stopped sending logs to cloudwatch after updating to 0.1.30 chart, https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit/0.1.30

had to revert to 0.1.21

[Breee]

Same problem, that holds also for elasticsearch output