AIP-38 Create new designs for Airflow 3

[pierrejeambrun]

Pages to design:

• New Homepage Dashboard
• DAGs list
• Assets page
• DAG Details

Many table/card view pages should follow the same designs (dag runs list, task instances list, user management, connections)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[bbovenzi]

Upgrading Cluster Activity to be the main dashboard for a new home page.

Stories:

• What has happened across my entire Airflow deployment recently?
• Can I quickly see any issues and drill down into them?

[bbovenzi]

Card-style view of the DAGs list page.

Users can still toggle to a more condensed table view. A few pieces here are still subject to change.

Stories:

• Show me a list of my dags with a summary of their activity
• We have a big org. I want to filter the list to just the specific dags I care about.
  

[bbovenzi]

Advanced queries for list views:

[bbovenzi]

New vertical navigation with subnav and menu options

[potiuk]

Slick!

[bbovenzi]

DAG Details

Stories:

• Show me an overview of this DAG's activity
• Highlight failed task instances and let me quickly get to their logs

We're going to make that top stats bar smaller and remove "Slow Tasks"

[bbovenzi]

DAG Details, view list of runs with a bar chart to help as a date range selector:

[bbovenzi]

DAG Details, list of tasks with their own graph of recent task instances:

Stories:

• Show me a list of my tasks with quick history of their recent instances.

[bbovenzi]

DAG Run Details, with the ability to see task logs for the entire run at once (we need to see if this is actually feasible to load so many logs at once)

[bbovenzi]

Task details to see how a task has performed across instances, same as a DAG overview.

[bbovenzi]

Task Instance details with logs:

[bbovenzi]

Graph, Grid and Gantt open up in a full-page modal that can be accessed from any Dag/Run/Task page. The modal also has a right hand drawer for information on the selected dag/run/task.

Motivation:

• Grid/Gantt/graph can all be complicated views. Give them the space of a full-page modal but still include a drawer for some details

[bbovenzi]

Logs and graph side-by-side:

[potiuk]

Nice :)

[jscheffl]

New vertical navigation with subnav and menu options

I don't know if this issue is the right way to discuss. let me know if there are other means...

I'd propose to move the XComs to "Browse..." as actually it is a legacy that you browse XComs in the Admin menu... but usually this is not an admin-specific thing.

[jscheffl]

Card-style view of the DAGs list page.

Users can still toggle to a more condensed table view. A few pieces here are still subject to change.

As we might not have a "Multie Team" setup in most environments... and as usually you have more than a hand-full of DAGs - can we foresee an option that a user can mark DAGs (on his profile) like "Favorite DAGs" which are displayed first before a long long list? (to focus on). At least we have many users who are mainly interested in 3-5 of 100 DAGs.

[jscheffl]

DAG Run Details, with the ability to see task logs for the entire run at once:

One thing that botheres me since a long time and is contained in the Lorem-Ipsum as well - there are always 3 lines where the Log is sourced from. Nice for troubleshooting bust most users are confused by this and wonder why link is most cases noch loading when clicking (because internal protected S3 backend, not public exposed...) - logs should focus on valueable content, in the summary maybe the last N lines and not the first lines.

[jscheffl]

Graph, Grid and Gantt open up in a full-page modal that can be accessed from any Dag/Run/Task page. The modal also has a left hand drawer for information on the selected dag/run/task.

I very much like the new views. What many users fail but is a common modelling pattern is to use mapped tasks. This usually needs a few more clicks to find and navigate down. Is there a way to have a direct drill-down also on mapped tasks on the new detail views? (As if it is vitually a task group with a set of tasks from UI perspective, just the volume is flexible per run).

[bbovenzi]

@jscheffl

• Good ideas on logs and xcoms. Both make sense to me
• We don't actually have any user settings saved except in localstorage which is why, for now, we have the "Recently viewed Dags" section in the homepage dashboard. Happy to figure out a proper user favorites though.
• Yes, any failed mapped task instance should have quick links to just like any other TI. So far I think we'll keep the existing UX of adding a Mapped Tasks tab to the Task Instance details page/drawer. In the case of the drawer, you can see the graph and the list of mapped tasks within the same view. If we manage to show logs across a whole dag run, perhaps we can show logs across a Task Group or all mapped tasks.

[kaxil]

Loving the new views

[ldacey]

Looks great. Are there any new views for Datasets/Assets in the works as well?

I have been splitting my DAGs up, one (or more) for extracting raw data and the other for processing/transformation. Right now, the dataset view shows a lot of dags instead of just the inlets/outlets for the DAG I am looking at (sometimes for other clients so it is not a view i can use while sharing my screen).

(extract DAG):
list_files (source dataset) -> extract_files (raw dataset) -> trigger_dag (outlet = raw dataset)

(transform DAG):
scheduled on raw dataset

transform_files (bronze dataset) -> deduplicate (silver dataset) -> finalize (gold dataset)

It would be nice to be able to see the lineage from the DAG if possible. So if I am within the transform DAG then I would see something like source > raw > bronze > silver > gold and be able to click on a link to the raw DAG.

[cmarteepants]

@ldacey they are in the works! we are slotted to work on them once we wrap up the dag view designs. agreed! this is something we want to show in the new asset views

[pierrejeambrun]

Really nice! 🎉

[bbovenzi]

Backfills:

See more button lists all possible Dag actions:

"Run Backfill" opens a modal:
(Need to add "dry_run" response here of how many runs will be rerun or created based on backfill options)

Add a banner to the Dag page when there is an active backfill

Tab to list all backfills:

Clicking on a singler backfill shows details about it like all dag runs its affected.

[ausiddiqui]

Task details to see how a task has performed across instances, same as a DAG overview.

Task run duration would be a great bar graph, and having the average over the x number of runs displayed would be awesome.

[cmarteepants]

@ausiddiqui not sure if it's just me, but I can't see your screenshot. Agree conceptually that task run duration would make a great bar graph!