Fix Outlet Event Extra Data is Empty in Task Instance Success Listener

[sjyangkevin]

Close: #52574

As discussed in #52574 , the outlet event extra data is empty because the task runner does not commit the asset changes to the database before calling the callbacks and the listeners. Also, every call to the get_template_context method create and return a new Context object, which might not persist the changes made during the task execution. Therefore, a method to fix this issue is to add a cache to the get_template_context method, ensuring all calls are operating on the same context object.

Since RuntimeTaskInstance is a unhashable type, @functools.cache doesn't work in this case. As an alternative, we probably can convert the context into a property, and the runtime task instance will operate on the property context. However, this implementation breaks the pre-commit hook check-template-context-variable-in-sync.

airflow/scripts/ci/pre_commit/check_template_context_variable_in_sync.py

Lines 81 to 88 in d4f4b22

	# Extract keys from the main `context` dictionary assignment
	context_assignment = next(
	stmt
	for stmt in fn_get_template_context.body
	if isinstance(stmt, ast.AnnAssign)
	and isinstance(stmt.target, ast.Name)
	and stmt.target.id == "context"
	)

Any feedback regarding how to improve the current implementation for fixing the issue would be appreciated.

manual functional testing

Before making the change, on log line 9, the bottom of the screenshot. The extra field is empty

After making the change, on log line 9, the bottom of the screenshot. The extra field contains the asset event extra.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[sjyangkevin]

Hi @uranusjr , would you mind have a look whenever you have time? Just want to make sure the fix is on the right direction and would appreciate if you have any feedback. thank you in advance!

[sjyangkevin]

Hi @ashb , I made the updates to the test cases, and also attempted to update the ci pre-commit check logic. Would appreciate if you could have a look at it and feel free to provide further feedback.

[Lee-W]

Overall looks good to me, but would appreciate a second pair of eyes

[Lee-W]

Mostly good, I would appreciate it if we could have someone more familiar with task-sdk to take another look, as this changes the behavior of context, which could have a huge impact.

@amoghrajesh @kaxil @ashb

[dygger]

Hi @sjyangkevin,

Thanks for your fix — I’ve applied it in my environment and everything is working smoothly now. Really appreciate your effort!

Looking forward to seeing this included in the next Airflow release.

[sjyangkevin]

Hi all, I would like to follow up on the PR. I think the major concern here probably is around the impact of the context; recently, I had very limited bandwidth, but will try to get back to this in the upcoming week or two, to further investigate the impact of the change. Thank you for the time to review the PR.

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[sjyangkevin]

Hi @Lee-W , thanks for looking into it , sorry I didn't get the chance yet to make the updates suggested by @uranusjr due to very limited bandwidth in these few weeks;

I am wondering if we still need to make that change, if so, I would like to push that change by around end of the week. thanks!

Let’s add a comment on this attribute to mark this as only implementation detail for caching get_template_context. And/or maybe rename this to _cached_template_context?