support retry multiplier parameter #56866
Conversation
steveahnahn
requested review from
XD-DENG,
amoghrajesh,
ashb,
bolkedebruin,
kaxil and
uranusjr
as code owners
ashb
left a comment
ashb
left a comment
There was a problem hiding this comment.
Do we need a new parameter for this, or could we allow retry_exponential_backoff to be set to an number instead to control it? (retry_exponential_backoff=True -> 2), retry_exponential_backoff=False -> 0, so its still falsey.
I am also tempted to say this should be a new ser dag version.
steveahnahn
requested review from
bugraoz93,
ephraimbuddy,
jason810496,
pierrejeambrun,
rawwar and
shubhamraj-git
as code owners
steveahnahn
requested review from
bbovenzi,
jscheffl and
ryanahamilton
as code owners
airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml
Show resolved
Hide resolved
|
One last moment idea... should we also add something to |
Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com>
steveahnahn
force-pushed
the
support-retry-multipler-parameter
branch
from
129ff5c to
bc657ea
Compare
|
Thanks for the suggestion, I looked into this but retry_exponential_backoff is a Dag operator parameter and not an airflow.cfg configuration setting therefore I think it wouldn't apply here. That said, I did find a bug that was addressed in bc657ea. When converting booleans to floats, float(True) returned 1.0 instead of the original default 2.0 multiplier. This would have completely broken exponential backoff for existing Dags using retry_exponential_backoff=True |
Ah, yeah 🤦 ypu are right is not a config entry :-D |
* support retry multiplier parameter * new param in serialization schema.json * fix test: retry multipler param matches schema default * replace back param with multipler * missed type hints * prek regen type hints * Update airflow-core/src/airflow/models/taskinstance.py Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com> * newsfragment addition & backwards compatability * fix ci/cd rebase prek --------- Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com>
* support retry multiplier parameter * new param in serialization schema.json * fix test: retry multipler param matches schema default * replace back param with multipler * missed type hints * prek regen type hints * Update airflow-core/src/airflow/models/taskinstance.py Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com> * newsfragment addition & backwards compatability * fix ci/cd rebase prek --------- Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com>
* support retry multiplier parameter * new param in serialization schema.json * fix test: retry multipler param matches schema default * replace back param with multipler * missed type hints * prek regen type hints * Update airflow-core/src/airflow/models/taskinstance.py Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com> * newsfragment addition & backwards compatability * fix ci/cd rebase prek --------- Co-authored-by: 김영준 <youngjun.kim@kurlycorp.com>
5 participants
Summary
Add configurable retry_delay_multiplier parameter to BaseOperator allowing users to customize the exponential backoff multiplier for task retries (previously hardcoded to 2).
Solution
Testing
Tested with breeze and unit tests with screenshots below:
Related Issues
Closes #56537