Skip to content

Conversation

@insomnes
Copy link
Contributor

This PR adds the ability to pull XCom when passing multiple map_indexes on RuntimeTaskInstance.

  • Update description in doc string
  • Pull XCom for each value from the product of all task ids and map indexes
  • Fix None case for task id
  • Expand original tests

closes: #46665


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@insomnes
Copy link
Contributor Author

insomnes commented Feb 13, 2025

I am still not sure if this is the intended implementation or not, but it is limited by the current execution API restriction of 1 XCom result in response. I didn't touch it to not break anything.

Prints from original DAG example:

    @dag.task
    def xcom_pull(**context):
        ti: TaskInstance = context["ti"]

        # [1, 3, None, None, None, None]
        for x in [
            ti.xcom_pull(task_ids=["identity", "foo", "plusfive"], map_indexes=[0, 2]),
        ]:
            print(x)

@insomnes
Copy link
Contributor Author

If this PR is not in the right direction I am ok with full rework. I just need to understand what was planned for this feature then.

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch from 42bea72 to dbb0021 Compare February 18, 2025 18:18
@amoghrajesh
Copy link
Contributor

Hello @insomnes, apologies for the delayed review. Taking a look at it.

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial set of comments, WDYT @ashb?

@insomnes
Copy link
Contributor Author

@amoghrajesh thanks for the review! No worries about the delay!
I appreciate your time and I’m glad I could contribute to Task SDK development.

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch 2 times, most recently from 2fdc2a9 to 6a31d6d Compare February 21, 2025 18:14
@insomnes
Copy link
Contributor Author

Up. Would be nice to know what's next.

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch 4 times, most recently from 80488d5 to f73609f Compare February 27, 2025 15:31
@insomnes
Copy link
Contributor Author

Up. If this PR should wait for something else or other solution should be chosen, than it would be nice to have some update about it.

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch 4 times, most recently from a2a9c84 to d8d4a15 Compare March 4, 2025 12:09
@insomnes insomnes force-pushed the xcom-pull-map-indexes branch 3 times, most recently from 521d251 to fa61af3 Compare March 8, 2025 17:30
@insomnes insomnes force-pushed the xcom-pull-map-indexes branch from fa61af3 to 66e6dfe Compare March 10, 2025 08:44
@amoghrajesh amoghrajesh requested a review from ashb March 12, 2025 06:13
@phanikumv phanikumv requested a review from kaxil March 12, 2025 07:01
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point later, we should just maybe expand the API endpoint to optionally get results for multiple map indexes, fine for now!

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patience @insomnes, its taken long to get a review here 🗡️

Few nits, preemptively approving once those are handled.

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch from 7a21a50 to b4c1de8 Compare March 13, 2025 15:21
@insomnes
Copy link
Contributor Author

Thank you @kaxil and @amoghrajesh for the review.

its taken long to get a review here 🗡️

No problem! I just got a bit lost on the best way to work around the current restrictions of the API, because I don't have a full picture.

I've applied the suggestions as is so I resolved the comments for them

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch 3 times, most recently from 12aee79 to 282ab87 Compare March 16, 2025 12:49
@insomnes
Copy link
Contributor Author

I've resolved two comments, they were properly fixed, I believe. Still left the thread with the API limitations discussion unresolved, because I am unsure if it is appropriate to do it myself in this case. Other threads were simple fixes.

@insomnes insomnes force-pushed the xcom-pull-map-indexes branch from 282ab87 to 6fbf1ff Compare March 18, 2025 22:27
@insomnes
Copy link
Contributor Author

Rebased on recent changes to task_runner.py.

@amoghrajesh
Copy link
Contributor

@insomnes i have rebased it again, once green, will merge, thanks for your patience

@insomnes
Copy link
Contributor Author

@insomnes i have rebased it again, once green, will merge, thanks for your patience

Thank you, checks are green now.
No problem. I am happy to help with the airflow development!

@amoghrajesh amoghrajesh merged commit 7ceb9c8 into apache:main Mar 19, 2025
61 checks passed
@insomnes insomnes deleted the xcom-pull-map-indexes branch March 19, 2025 08:34
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
)

* Add ability to pull XCom when passing multiple map_indexes

* Add comment about LazyXComSequence and orignal description

* Apply review comments

* Update task_sdk/src/airflow/sdk/execution_time/task_runner.py

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

* Fix review comments

---------

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
shubham-pyc pushed a commit to shubham-pyc/airflow that referenced this pull request Mar 22, 2025
)

* Add ability to pull XCom when passing multiple map_indexes

* Add comment about LazyXComSequence and orignal description

* Apply review comments

* Update task_sdk/src/airflow/sdk/execution_time/task_runner.py

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

* Fix review comments

---------

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
)

* Add ability to pull XCom when passing multiple map_indexes

* Add comment about LazyXComSequence and orignal description

* Apply review comments

* Update task_sdk/src/airflow/sdk/execution_time/task_runner.py

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

* Fix review comments

---------

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Not able to pull XCom when passing multiple map_indexes

4 participants