Jinja does not find XCom value with dynamic mapping for Airflow 3

[sveyret]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

We are using an operator in a dynamic mapping. One of the static values, defined in partial is a template, using Jinja to extract an XCom value. This XCom value is not found by Jinja and replaced with "None".

This was working with version 2.x

What you think should happen instead?

We expect the template to be replaced appropriately, as it was in Airflow 2.x.

How to reproduce

In the MVE below, the show task is displaying None instead of Show this:

from datetime import datetime
from textwrap import dedent

from airflow import DAG
from airflow.decorators import task
from airflow.providers.standard.operators.bash import BashOperator


@task
def build_something():
    return "Show this"


@task
def build_multiple():
    return [{"message": "Also this"}, {"message": "Also that"}]


with DAG(
    "mve-xcom-jinja",
    default_args={
        "depends_on_past": False,
    },
    schedule=None,
    start_date=datetime(2025, 1, 1),
    catchup=False,
) as dag:
    something = build_something()

    multiple = something >> build_multiple()

    _ = multiple >> BashOperator.partial(
        task_id="show",
        bash_command=dedent(
            """
            echo "{{ task_instance.xcom_pull(task_ids='build_something') }}"
            echo "$message"
            """
        ),
    ).expand(env=multiple)

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[amoghrajesh]

@sveyret its along the line of issues I am currently focusing on, let me self assign it to myself. Will pick it up next week.

cc @TJaniF is this the issue you were mentioning in #50636 ?

[TJaniF]

@amoghrajesh yes, I was pulling the xcom from context in a task flow task, not using jinja, but I assume this is the same underlying issue with XCom of dynamic tasks

[phanikumv]

Can we get this done for 3.0.2 @amoghrajesh ?

[amoghrajesh]

@phanikumv connected with TP on this and this can be fixed by adding a simple hook that implements / uses #50017 as a baseline. #50017 is targeted for 3.1.0 so this should probably be present in 3.1.0 too. @uranusjr do you think we should be pulling in #50017 for 3.0.2?

[uranusjr]

I think #50011 is actually in 3.0.1. Do you mean #50117 instead? (If so, yes I think we can put it in 3.0.2 if we want to.)

[amoghrajesh]

Right sorry, my bad. Its #50117. Let's try and put #50117 into 3.0.2 in that case