Fix: Grid and Graph Rendering Issues in Airflow 3.0 for Dynamic DAGs

[arunangshu01]

Related Issues -

1. closes: Airflow 3.0.2: Airflow Grid UI not rendering. #51955
2. closes: Airflow 3.0.2: Airflow Graph UI not rendering task state colors. #51954

This PR addresses a consistent issue observed during recent tests involving dynamic DAGs in Airflow 3.0. Specifically, when a variable is updated within an ongoing task, the DAG version used for rendering the graph view incorrectly falls back to the initial DAG version instead of the latest one.

As a result, the graph colors are not properly rendered, making it difficult to trace the current state and flow across different stages of the DAG run. This fix ensures that both the grid and graph views correctly reference the latest DAG version (in descending order), improving visibility and debugging during pipeline execution.

Contributors are welcome to review, test, and suggest improvements.

Thank you!

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[bugraoz93]

Are these related? Maybe we can prevent duplicate work
#53087

[arunangshu01]

@bugraoz93 - Thanks for the comment. I’ll review the PR, pull the changes, and check if the issue I raised has been resolved.

[arunangshu01]

@bugraoz93 - Thank you for resolving the grid issue - I have reviewed the latest main branch as well as your PR, and things are looking good on that front.

I also tested the 3.0.3rc5 pre-release for my dynamic DAG use case. While the grid view now renders properly, I’m still noticing issues with task instance colors in the graph view - they don’t reflect the expected states.

I would be happy to test a newer pre-release once those fixes are in, especially to verify whether the grid and graph color rendering are fully resolved.

If helpful, I’m available for a quick 1:1 call to walk you through the issue (though I won’t be able to share actual code due to organizational confidentiality policies). :)

Thanks again for the prompt fix and ongoing support !

[mpunch1]

Are these related? Maybe we can prevent duplicate work #53087

@bugraoz93 One of the issue is related to that, but for the dynamic DAG when the version changes while the DAG is in running state the new graph is being generated but the state Colour are not rendering as the api is failing

[pierrejeambrun]

@mpunch1 @arunangshu01 can you double check on latest RC (rc5), also can you provide a minimal dummy DAG that reproduces the problem?

[arunangshu01]

from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import Variable
import pendulum
from datetime import timedelta
import time
import ast

default_args = {
    'owner': 'airflow',
    'retries': 5,
    'retry_delay': timedelta(minutes=1),
}

def set_list_variable(**kwargs):
    Variable.set('sample_list', str(list(range(1, 1500))))

def get_list():
    val = Variable.get('sample_list', default='[1]')
    return ast.literal_eval(val) if isinstance(val, str) else val

def iterator_task(i, **kwargs):
    print(f"Processing item {i}, sleeping for 2 seconds")
    time.sleep(2)

def iterator_task1(i, **kwargs):
    print(f"Processing item {i}, sleeping for 2 seconds")
    if isinstance(i, str):
        raise Exception
    time.sleep(2)

def final_task(**kwargs):
    print("All iterator tasks are done.")

with DAG(
    'dynamic_sequential_dag',
    default_args=default_args,
    description='DAG with variable-driven dynamic tasks (classic loop)',
    schedule=None,
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    catchup=False,
    tags=['example'],
) as dag:

    t1 = PythonOperator(
        task_id='set_list_variable',
        python_callable=set_list_variable,
    )

    t2 = PythonOperator(
        task_id='task_2',
        python_callable=iterator_task,
        op_args=[2],
    )

    tasks = []
    for i in eval(Variable.get('sample_list', default='["dummy"]')):
        task = PythonOperator(
            task_id=f'iterator_task_{i}',
            python_callable=iterator_task1,
            op_args=[i],
            retries=5,
        )
        tasks.append(task)

    t3 = PythonOperator(
        task_id='final_task',
        python_callable=final_task,
    )

    t1 >> t2 >> tasks >> t3

@bugraoz93, @pierrejeambrun - Here you go with a sample dag code.

[bugraoz93]

Are these related? Maybe we can prevent duplicate work #53087

@bugraoz93 One of the issue is related to that, but for the dynamic DAG when the version changes while the DAG is in running state the new graph is being generated but the state Colour are not rendering as the api is failing

I just followed up. Thanks to Pierre! :) Also, thanks for your PR!

@bugraoz93 - Thank you for resolving the grid issue - I have reviewed the latest main branch as well as your PR, and things are looking good on that front.

Thanks for example!

[bugraoz93]

Is this DAG failing?

[mpunch1]

Is this DAG failing?

yes the DAG either fails(even after the retries) or it get parsed before reaching the task and the new DAG is graph is rendered but the task state colors are not reflected because the api fails.
The error i have mentioned in the #53087
I see that the routes have been restructured and there is significant change in grid routes queries will give it a try with latest main code.

Thanks @bugraoz93

[bugraoz93]

Is this DAG failing?

yes the DAG either fails(even after the retries) or it get parsed before reaching the task and the new DAG is graph is rendered but the task state colors are not reflected because the api fails.
The error i have mentioned in the #53087
I see that the routes have been restructured and there is significant change in grid routes queries will give it a try with latest main code.

Thanks @bugraoz93

Thanks that would be amazing @mpunch1!

[pierrejeambrun]

Thanks @arunangshu01 for the reproducible example.

cc: @dstandish This might be related to the recent grid optimization / refactoring.

[pierrejeambrun]

I see that the routes have been restructured and there is significant change in grid routes queries will give it a try with latest main code.

Let us know your findings, thanks

[arunangshu01]

Hey @bugraoz93 @pierrejeambrun - We are seeing the same issue in 3.0.3.
We have also seen that the 3.0.3 tag is not aligned with the main branch.

Can you tell me when can we expect the new version to arrive so that we can test and check it out?

[potiuk]

Hey @bugraoz93 @pierrejeambrun - We are seeing the same issue in 3.0.3. We have also seen that the 3.0.3 tag is not aligned with the main branch.

Can you tell me when can we expect the new version to arrive so that we can test and check it out?

You can test even today in the main branch - following contribution guide. Otherwise (like everyone else) subscribe to devlist and when release candidate will be prepared for testing you will get information and you will be able to test it. You can also follow our wiki and take part in the dev calls - tentatively Airflow 3.1 is planned to be released some time in September. But it's not firm or promised date - you need to follow devlist to find out.

You will find links and more information on how to subscribe etc in "community" tab on https://airflow.apache.org

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[pierrejeambrun]

up

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[pierrejeambrun]

First issue was closed by another PR #51955

Second issue for colors I cannot reproduce with your dag example. Bug has most likely been fixed, this is what I get with your reproducing dag on main(#51954)