Feature(call-back): Enhance DAG-level callback context with enriched metadata and test coverage #51949
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kiran2706
requested review from
XD-DENG,
amoghrajesh,
ashb and
kaxil
as code owners
shahar1
reviewed
Jun 20, 2025
Contributor
shahar1
left a comment
shahar1
left a comment
There was a problem hiding this comment.
Please fix static checks :)
Contributor
Author
|
Working on it. |
uranusjr
reviewed
Jun 23, 2025
uranusjr
reviewed
Jun 23, 2025
uranusjr
reviewed
Jun 23, 2025
…metadata and test coverage
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
kiran2706
force-pushed
the
feature/improve-dag-callback-context
branch
from
c1185a1 to
9493126
Compare
Contributor
Author
|
Hey @uranusjr, We can't use context.update(task_instance=ti) since TypedDict.update() doesn't support keyword arguments — mypy throws an error:
So I've updated it to use a standard dict instead. Please re-review the changes. Thanks in advance. |
uranusjr
reviewed
Jun 24, 2025
Comment on lines
+86
to
+90
| ``{{ mark_success_url }}`` str | None |URL to mark the DAG run as successful in the Airflow UI. | ||
| ``{{ log_url }}`` str | None |URL to the log for the current DAG run or task instance. | ||
| ``{{ dag_run_url }}`` str | None |URL to the DAG run details page in the Airflow UI. | ||
| ``{{ end_date }}`` DateTime | None |The end date/time of the DAG run. | ||
| ``{{ max_tries }}`` int | None |The maximum number of tries for the task instance. |
Member
There was a problem hiding this comment.
I think this should be removed now they are not being added
uranusjr
reviewed
Jun 24, 2025
Comment on lines
+1365
to
+1366
| success_tis.sort(key=lambda x: x.end_date, reverse=True) | ||
| last_relevant_ti = success_tis[0] if success_tis else None |
Member
There was a problem hiding this comment.
This can be something like
last_relevant_ti = max(success_tis, ...) if success_tis else Noneinstead.
I kind of wonder if we can even avoid building the list at all.
Also is simply sorting by end_date correct? Especially with trigger_rule, the last success/failed ti might not necessarily be the ti that causes the dag run to be marked as success/failed. Can you check the logic in 2.x to see how the ti is selected?
uranusjr
reviewed
Jun 24, 2025
Comment on lines
+2072
to
+2073
|
|
||
| return prefix + ">" |
uranusjr
reviewed
Jun 24, 2025
Comment on lines
+89
to
+93
| "dag_run_url", | ||
| "end_date", | ||
| "log_url", | ||
| "mark_success_url", | ||
| "max_tries", |
2 tasks
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 23, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 23, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 23, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 24, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Jul 24, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
aws-airflow-bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 24, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
ferruzzi
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Aug 7, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
kaxil
added a commit
that referenced
this pull request
Aug 9, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to #53058 Fixes #52824 Fixes #51402 Closes #51949 Related to #53654 Related to #53618 (cherry picked from commit ef80507)
fweilun
pushed a commit
to fweilun/airflow
that referenced
this pull request
Aug 11, 2025
This ensures DAG callbacks receive the same rich context as task callbacks, improving consistency and providing access to template variables and macros similar to Airflow 2. This has been a blocker for few users similar to apache#53058 Fixes apache#52824 Fixes apache#51402 Closes apache#51949 Related to apache#53654 Related to apache#53618
54 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR enriches the context passed to DAG-level callbacks (on_success_callback and on_failure_callback) to better align with the structure and metadata typically available in task-level callbacks. While this logic is currently triggered via execute_callbacks=True during tests, it is structured to reflect what should be expected in production use cases as well.
Related
Closes: [#51402]