Grid view scaling issues

[jedcunningham]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The grid view doesn't scale well as either task count or dag version count grows.

Imagine you have 100k tasks in the dag, and 10 runs...

What you think should happen instead?

After having a brief chat with @bbovenzi and @dstandish, we need to do some sort of pagination. It's slightly more complicated when you consider the structure (and thus order) can change a between runs, and accounting for things like task groups and mapped tasks, but those should be surmountable problems.

One idea is to continue using the order from the latest dag version, then go determine the state etc of the n tasks from earlier runs, regardless of where in the older runs those tasks were.

With pagination implemented, we can do things like infinite scroll in the UI too.

How to reproduce

Create a few runs on a really large dag.

Operating System

macos

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[bbovenzi]

+1 to paginating grid data by the first ~50 tasks based on the topological sort of the latest dag version. This also reduces the total number of grid items the UI will try to render at once

Other possible solutions:

• making grid data only return task_id and state for a quick render and then fetch the specific TI information
• move task group overall state calculation to the frontend in a way that doesn't block loading all other tasks

But paginating grid data helps improve the API and UI performance so let's explore that first