KubernetesPodOperator do_xcom_push=True and deferrable=True does not push return_value to xcom #58796
Description
Apache Airflow Provider(s)
cncf-kubernetes
Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==10.10.0
Apache Airflow version
2.11.0
Operating System
Debian 12
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
What happened
We use do_xcom_push = True on KubernetesPodOperator like below, we also set deferrable=True
write_xcom = KubernetesPodOperator(
namespace="default",
image="{{ macros.rca_plugins.ecr_repo() }}/roscommon/rcapvt:{{ var.value.rcapvt_image_tag }}",
cmds=["sh", "-c", "mkdir -p /airflow/xcom/;echo '[5,6,7,8]' > /airflow/xcom/return.json"],
name="write-xcom",
do_xcom_push=True,
deferrable=True,
on_finish_action="delete_pod",
in_cluster=True,
task_id="write-xcom",
get_logs=True,
)
On apache-airflow-providers-cncf-kubernetes==10.9.0, this all working fine, I can see return_value in xcom
But once I upgrade to version 10.10.0, this do_xcom_push functionality no longer works, I cannot see return_value in xcom.
It works in 10.10.0 in non deferrable mode, in 10.9.0 deferable or non-deferable works fine.
What you think should happen instead
do_xcom_push should work either in deferrable or non-deferrable, return_value should get populated in xcom
How to reproduce
Create a dag with below task:
write_xcom = KubernetesPodOperator(
namespace="default",
image="alpine",
cmds=["sh", "-c", "mkdir -p /airflow/xcom/;echo '[1,2,3,4]' > /airflow/xcom/return.json"],
name="write-xcom",
do_xcom_push=True,
deferrable=True,
on_finish_action="delete_pod",
in_cluster=True,
task_id="write-xcom",
get_logs=True,
)
Run it and you will not see see return_value in xcom
Anything else
I check PR #57531
on pod.py file
https://github.com/apache/airflow/pull/57531/files#diff-253d1c55d88669a074ae3cbfa0a79a2dfc811277cdb32642a01a9dbaddb82f30
it delete line 946:
return xcom_sidecar_output
I think this is what broke the functionality.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct