Showing Task State in Gantt Chart?

[yxiao1996]

Description

Hi team,

Recently I've been working on using Airflow to orchestrate some of my teams ETL jobs. Lately I run into some performance issue and noticed that the Gantt chart feature is a pretty handy tool for understanding long running DAG runs. I observed large gaps between tasks in the Gantt chart, showing there's a resource bottleneck. However, it seems like the Gantt chart only tracks "running" state of the tasks, so I was not able to fully pin-point whether the bottleneck is the worker(tasks being "queued" for a long time) or the scheduler(tasks being in "no_status" for a long time). So I'm thinking about if airflow could track different task state in the Gantt chart, it would be easier for users to troubleshoot similar issues.

Please educate me if my understanding is wrong.

Thanks,
Yu

Use case/motivation

By having task state tracking in Gantt chart, it would be easier for airflow users to troubleshoot performance issue in DAG's.

Related issues

n/a

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[bbovenzi]

We do have queued_dttm for task instances, so that shouldn't be too hard to add. Otherwise, I think we'll need a new table to record task events like state changes?

[pierrejeambrun]

Hello @yxiao1996,

Good idea!

I see you are willing to contribute, assigning you :)

[yxiao1996]

thanks for the comments. wip....

[yxiao1996]

Hi team, I looked into the code a little bit, want to check with you about the general approach before making the change.

Seems like I should be able to retrieved the queued_dttm field from the ti objects here? If that is the case, looks like I can add queued_dttm field into task_dict, which would be passed to the front-end. Then I can draw new boxes in gantt.js around here with box width defined by queued_dttm and task start_date.

Am I making sense?

[bbovenzi]

Yes, that sounds about right. And the class will just be queued instead of the task state. You can see how the current boxes are created here. Reach out if you have any other questions!

[yxiao1996]

Hi,

I think I'm getting close to put up a PR. I noticed that the queued_dttm is already passed to the UI through this line so I'm only changing the javascript. I'm curious are there any tests I need to created for this new feature? I don't see on the surface there's any tests for these javascripts. Also, is there a template I need to follow for PR?

Thanks,
Yu

[yxiao1996]

Here's a snapshot of this feature in my dev environment. Basically I'm drawing these gray boxes for tasks instances with non-null queued_dttm, also add a tooltip on the queued state box to provide related info. Curious do you guys have any comments on how this should look? @bbovenzi @pierrejeambrun

[bbovenzi]

Looks great! I'd say open a PR so we can can a closer look. If it's not fully ready, open it up as a draft.

[eladkal]

Looks nice!
BTW would be intresting as a followup to see what we can do about #17487

[bbovenzi]

Closed by #28686

But a future improvement could be to use Task Reschedules in Gantt as explained in #17487