Support configurable multiplier for retry exponential backoff #56537
Description
Description
I love using Airflow to build workflow logic, but I've hit a limitation with the current retry options.
Right now, I need to implement a customizable retry logic — specifically, I want to set a configurable multiplier for the retry exponential backoff, but the multiplier is currently hardcoded to 2 in Airflow.
def next_retry_datetime(self):
...
min_backoff = math.ceil(delay.total_seconds() * (2 ** (self.try_number - 1)))
...So I suggest adding a new parameter such as retry_delay_multiplier to allow users to configure the exponential backoff.
# like this!
default_args = {
'depends_on_past': False,
'retries': 2,
'retry_exponential_backoff': True,
'retry_delay': timedelta(minutes=4),
'retry_delay_multiplier': 5,
}It will retry after 4 min, then 20 min, and then 100 min.
Use case/motivation
Many data pipelines or APIs have different retry requirements: the fixed exponential backoff with a multiplier of 2 is sometimes too aggressive or too slow depending on the job's nature. If Airflow provides a configurable multiplier for exponential backoff, users can fine-tune retry intervals for different workloads. This is especially useful for critical external integrations, error-prone batch jobs, or systems with rate limits, where controlling the growth rate of delay is essential for stability, throughput, and SLA adherence.
Customizing the multiplier would make Airflow's retry system much more versatile and suitable for a wider variety of operational scenarios.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct