Add support for timeout-based Retry_Limit

[pkqsun]

Is your feature request related to a problem? Please describe.
Discussions link: #10360

Refer to https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configure-retries , there are 3 types of values for scheduler Retry_Limit: N, no_limits and no_retries.

For our case, the output is Loki and we set Retry_Limit with no_limites in order to not lost any logs when Loki is down.
In Loki site, we allow to accept these old logs within 2 days by setting proper value of ingester.max_chunk_age.
Now our scenario is that in a rainy case, we manually make Loki to offline and let fluent-bit to buffer/cache local chunk/logs more than 2 days, then online the Loki.
With the scheduler of retry, local chunks will be flushed to Loki one by one with each dedicated task_id.
However, for these timestamps that older than 48h, Loki will reject to accept them by reporting "write operation failed, older acceptable timestamp is xxxx" error. For fluent-bit, the flush action is not successful and will loop in the endless retries.

Describe the solution you'd like

Above is the actual issue for us, when Loki could be back within 48h, all local buffered chunks could be flushed to Loki with no_limits successfully, but when it is larger than 48h, some logs will be sent to Loki endless and Loki will always reject them.

Therefore, I would like to propose another values of Retry_Limit which is a timeout-based value.

For example, when Retry_Limit is set to 24h, then for one specific retry action will be same as no_limits from start, but when the retry action(maybe one task_id) reached 24h, it will stop the retry.

Describe alternatives you've considered

Additional context

Appreciate for any comments.

[duj4]

As @pkqsun mentioned, this feature would cease the endless retry in Fluent-Bit. Otherwise, end user has to delete those obsolete chunks and restart the service to cease it.

[duj4]

hi @patrick-stephens , could you please help check this request?

[agup006]

This should be a separate parameter (Drop_Chunks_Older_Than X) or something similar.

[duj4]

This should be a separate parameter (Drop_Chunks_Older_Than X) or something similar.

Thanks @agup006 for the suggestion, yes, if there is such parameter added, it would better for us to control the retry.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[duj4]

Please keep this open.