-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Fix XCom deserialization for mapped tasks with custom backend #50687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
amoghrajesh
merged 1 commit into
apache:main
from
astronomer:fix-xcombackend-mapped-tasks
May 16, 2025
Merged
Fix XCom deserialization for mapped tasks with custom backend #50687
amoghrajesh
merged 1 commit into
apache:main
from
astronomer:fix-xcombackend-mapped-tasks
May 16, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kaxil
approved these changes
May 16, 2025
Member
kaxil
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good find
eladkal
approved these changes
May 16, 2025
vatsrahul1001
approved these changes
May 16, 2025
github-actions bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
May 16, 2025
…kend (apache#50687) (cherry picked from commit 7ebba78) Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
sanederchik
pushed a commit
to sanederchik/airflow
that referenced
this pull request
Jun 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #50636
What the problem is?
While using a dag that has tasks consuming xcoms from upstream tasks which are mapped and a custom xcom backend is configured, the downstream tasks would not get the "data" of the xcom or the value of the xcom, but would instead get the reference of it.
In case of object store backend, the downstream tasks were suffering with the "path" on the store of the xcom.
Problem
Due to some recent refactoring done by: #50011, the deserialize function on the custom xcom backend was not consumed and "BaseXcom" was consumed always leading to it doing the most basic deserialize -- (serde deser) and hence the retrurned object remained equal to the reference of it.
Testing
DAG used (same as the reported one)
generate_datapushes right stuff:filter_item_from_mapped_taskpushes right stuff:map index 0:
map index 1:
map index 2:
Push on s3 is as expected:

Consumed rightly by the
print_itemtask now:map index 0:
map index 1:

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.