Rearrange Dag details view

[bbovenzi]

Problem:

• The Dag Details page layered information in too many separate pages that required too many clicks to navigate around. Especially if one went to the graph or grid modal and then closed it

Solution:

• "Flatten" the dag details page with three elements:

1. An improved breadcrumb header. Includes a header for each level. For a dag shows if it is paused or unpaused. For a dag run, shows its state. Always show the Trigger Run button
2. A left hand panel to navigate around a dag. The user can select either the grid or graph and can adjust the size of the panel to what works for them. We check the dag's default_view value and then persists the panel width and view selection in localStorage by dagId
3. A right hand panel to show details depending on what level the user has selected (Dag, Dag Run, Task, Task Instance, or Mapped Task Instance). This panel is responsive, so it should display useful information at any size.

Other features:

• Add state icons to the Grid view. Apply an opacity to successful tasks to make it easier to spot errant instances
• Truncate very long task ids in the task instances list so you don't have to scroll for forever

To Do in other PRs:

• Collapse/Expand all groups toggle
• Quick navigation between dag runs (Either by turning the dag run breacrumb into a dropdown or adding onto the Search Dags input)
• Handle selecting a mapped task instance without a map_index specified
• Zoom the graph to a selected task
• Grid/graph view filters
• Integrate gantt view

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[jscheffl]

Very cool.

One nit and... I started with Firefox on Ubuntu and the grid somehow does not populate. Is this a bug?

[bbovenzi]

Very cool.

One nit and... I started with Firefox on Ubuntu and the grid somehow does not populate. Is this a bug?

Are there any errors in the console? And if you look in the network tab for http://localhost:29091/ui/grid/{dag_id}?offset=0&limit=14&order_by=-run_after, does it exist and is there a response?

[jscheffl]

Are there any errors in the console? And if you look in the network tab for http://localhost:29091/ui/grid/{dag_id}?offset=0&limit=14&order_by=-run_after, does it exist and is there a response?

Ah, you are right... HTTP 500:

[jscheffl]

Are there any errors in the console? And if you look in the network tab for http://localhost:29091/ui/grid/{dag_id}?offset=0&limit=14&order_by=-run_after, does it exist and is there a response?

A different DAG is working better:

[bbovenzi]

Ah, you are right... HTTP 500: ![image](https://private-user-images.githubusercontent.com/95105677/415363640-

I tried the same dag and could replicate. But this is a bug on main, not due to changes to this branch.

[jscheffl]

Approve - under the assumption that you fix the three dots :-D

[tirkarthi]

Thanks for bringing back the grid view. I tried the PR locally. Couple of suggestions from my end.

1. The presence of titles dag, dag run, etc makes the breadcrumb little misaligned at first look personally. The Airflow 2 UI doesn't have titles and I think it will be good to remove them making the breadcrumb cleaner.

2. There could be a minimum width to the panel. Scrolling all the way to left causes the grid to render above task ids. Airflow 2 had some minimum width

3. There could be a full screen button for each panel so that views like graph and right panel can be toggled to utilize the full screen space. Airflow 2 had feature where one can click on a dagrun, select full screen to have only the column and go through all the details in the right pane for the dagrun.

4. Trigger dag doesn't refresh the grid. Clearing a run and marking the dagrun updates the breadcrumb but grid is not refreshed.

5. Next run after and schedule value being always displayed in the top right was useful in Airflow 2 irrespective of the view. This seems to be present only on selecting the dag page.

6. Task instance state color legend could also be presented.

[bbovenzi]

Thanks for bringing back the grid view. I tried the PR locally. Couple of suggestions from my end.

1. The presence of titles dag, dag run, etc makes the breadcrumb little misaligned at first look personally. The Airflow 2 UI doesn't have titles and I think it will be good to remove them making the breadcrumb cleaner.
2. There could be a minimum width to the panel. Scrolling all the way to left causes the grid to render above task ids. Airflow 2 had some minimum width
3. There could be a full screen button for each panel so that views like graph and right panel can be toggled to utilize the full screen space. Airflow 2 had feature where one can click on a dagrun, select full screen to have only the column and go through all the details in the right pane for the dagrun.
4. Trigger dag doesn't refresh the grid. Clearing a run and marking the dagrun updates the breadcrumb but grid is not refreshed.
5. Next run after and schedule value being always displayed in the top right was useful in Airflow 2 irrespective of the view. This seems to be present only on selecting the dag page.
6. Task instance state color legend could also be presented.

1. I was back and forth on this one. I heard a lot of comments from users that they didn't know what the breadcrumbs were so I wanted to figure out some way to make it more obvious.
2. Yes, I noticed that overflow. Its on my to-do list.
3. Yes, I think I'll do full screen in a follow up PR. This one was already getting big.
4. That is a bug I need to fix.
5. Hmm let me figure out where that could fit in the page. Not going to be for this PR.
6. Yes, in a follow up PR we will need a legend for task instance state icon+color

[tirkarthi]

Thanks @bbovenzi .

[bbovenzi]

Thanks @bbovenzi .

Did you play with resizing the panels? When this is merged. I think we'll need to fix our duration charts to stay a consistent size.

[tirkarthi]

Thanks @bbovenzi .

Did you play with resizing the panels? When this is merged. I think we'll need to fix our duration charts to stay a consistent size.

Yes, the x axis names didn't fit the chart and the chart was also compressed on smaller spaces not being very responsive. IIRC initially there was design to have 3 charts as a grid with remaining 2 charts requiring API for grouping and other idea to have a calendar/heatmap view of the task durations. Maybe on resize they can become a chart per row or to have the duration chart as the only chart for now till the other charts are finalized.

[ashb]

The presence of titles dag, dag run, etc makes the breadcrumb little misaligned at first look personally. The Airflow 2 UI doesn't have titles and I think it will be good to remove them making the breadcrumb cleaner.

How about reducing the line height on the breadcrumb label:

Current

Reduced height

(I also noticed that the height of the Task one doesn't match the DAG and Dag Run crumb)

Or playing with the vertical alingment of labels vs values:

[pierrejeambrun]

Grid data should be invalidated on actions (clear, mark as etc...) because for now the grid stay identical until we hard refresh after clearing for instance. Or even better auto refreshed.

I can't clear a task instance (error 500 but do not seem related to this PR, task sdk issue i think

Mark run as is missing the initial tasks table. (The initial dry-run call is not being send)

[bbovenzi]

The presence of titles dag, dag run, etc makes the breadcrumb little misaligned at first look personally. The Airflow 2 UI doesn't have titles and I think it will be good to remove them making the breadcrumb cleaner.

How about reducing the line height on the breadcrumb label:

Does this work for you?

[ashb]

Looks good. I think lets not sweat the small details too much in this PR; lets get this in for the first Beta as it's such a massive UX improvement over what is there right now!

[bbovenzi]

Looks good. I think lets not sweat the small details too much in this PR; lets get this in for the first Beta as it's such a massive UX improvement over what is there right now!

Agreed. I think its important to get this major change in. We'll get more feedback and can keep improving it. I don't want to leave this open for a week because I'm fiddling with css details here and there.