Skip to content

Conversation

@amoghrajesh
Copy link
Contributor

closes: #50636

What the problem is?

While using a dag that has tasks consuming xcoms from upstream tasks which are mapped and a custom xcom backend is configured, the downstream tasks would not get the "data" of the xcom or the value of the xcom, but would instead get the reference of it.

In case of object store backend, the downstream tasks were suffering with the "path" on the store of the xcom.

image

Problem

Due to some recent refactoring done by: #50011, the deserialize function on the custom xcom backend was not consumed and "BaseXcom" was consumed always leading to it doing the most basic deserialize -- (serde deser) and hence the retrurned object remained equal to the reference of it.

Testing

DAG used (same as the reported one)

from airflow.decorators import dag, task
from airflow.models.xcom import XCom

@task
def generate_data():
    print("XCom.__name__ --->", XCom.__name__)
    return [1, 2, 3]


@task
def filter_item_from_mapped_task(item):
    print("filter_item_from_mapped_task item --->", item)
    if item==1:
        return None
    return item


@task
def print_item(item):
    print("print_item item --->", item)


@dag(
    dag_id='test_bug_dag',
)
def test_bug_dag():
    data = generate_data()
    filtered_data = filter_item_from_mapped_task.expand(item=data)
    print_item.expand(item=filtered_data)

test_bug_dag()

generate_data pushes right stuff:

image

filter_item_from_mapped_task pushes right stuff:

map index 0:

image

map index 1:

image

map index 2:

image

Push on s3 is as expected:
image

Consumed rightly by the print_item task now:

map index 0:

image

map index 1:
image


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh amoghrajesh requested review from ashb and kaxil as code owners May 16, 2025 10:38
@amoghrajesh amoghrajesh added this to the Airflow 3.0.2 milestone May 16, 2025
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find

@amoghrajesh amoghrajesh added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label May 16, 2025
@amoghrajesh amoghrajesh merged commit 7ebba78 into apache:main May 16, 2025
89 checks passed
@amoghrajesh amoghrajesh deleted the fix-xcombackend-mapped-tasks branch May 16, 2025 16:40
github-actions bot pushed a commit that referenced this pull request May 16, 2025
…kend (#50687)

(cherry picked from commit 7ebba78)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@github-actions
Copy link

Backport successfully created: v3-0-test

Status Branch Result
v3-0-test PR Link

github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 16, 2025
…kend (apache#50687)

(cherry picked from commit 7ebba78)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
kaxil pushed a commit that referenced this pull request May 20, 2025
…kend (#50687)

(cherry picked from commit 7ebba78)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
potiuk pushed a commit that referenced this pull request May 21, 2025
…kend (#50687) (#50697)

(cherry picked from commit 7ebba78)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
kaxil pushed a commit that referenced this pull request Jun 3, 2025
…kend (#50687) (#50697)

(cherry picked from commit 7ebba78)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
sanederchik pushed a commit to sanederchik/airflow that referenced this pull request Jun 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-sdk backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

XComObjectStorageBackend in Airflow 3.0.1 Returns File Paths Instead of Deserialized Objects in Mapped Tasks

5 participants