Add "retry_on_failure" setting to filelog

[djaglowski]

User reports seeing the following error when using the logsCollection preset.

2023-06-27T20:26:11.105Z error consumerretry/logs.go:39 ConsumeLogs() failed. Enable retry_on_failure to slow down reading logs and avoid dropping. {"kind": "receiver", "name": "filelog", "data_type": "logs", "error": "data refused due to high memory usage"}

Likely the problem is that the filelog receiver is ingesting data at a faster rate than the collector can export. A new retry_on_failure configuration was recently added, which allows the the receiver to respond to backpressure by blocking ingesting until data is successfully consumed by the next component.

Additional context: https://cloud-native.slack.com/archives/C01N5UCHTEH/p1687898643893809

[TylerHelmuth]

Seems like a good idea. @djaglowski is the best practice to enable this feature by default?

[djaglowski]

Seems like a good idea. @djaglowski is the best practice to enable this feature by default?

@dmitryax may be better suited to answer this but I think likely we should.

[dmitryax]

Yes, I think we should enable it by default.

[husnialhamdani]

Hi, Can I work on this?