feat: add task upstream/downstream filter to Graph & Grid

[OscarLigthart]

Overview

We noticed that the lineage filter was missing in the Graph layout. At our company, the DAGs get quite complex, and this was a beloved feature by our users. In this PR, I try to have a crack at implementing it. I did my best to ensure that it feels similar to the old button in Airflow 2 in terms of its behaviour.

UI addition

See image:

When dropping down, a user can select the same options as before:

When selected, it shows at the top what filter is active:

From here, the user can navigate with the active filter. When reselecting this filter dropdown on another task focus, the user gets the ability to take this selected task as root task for the filter:

Please feel free to mess around with it!

Some notes from me on UX:

• I tried to make it feel natural, while also mimicking behaviour I saw in Airflow 2, but very open to hear suggestions on improving UX.
• I moved it from the task overview (where it was in Airflow 2) to Graph & Grid, as I feel like these are the components controlled by the filter, and it's more natural to control it from there.
• Implemented for both Graph & Grid, where filter selection persists while navigating.

Note to reviewer

I did some React back in the day, but it's a bit rusty. I'm here to contribute, but definitely also to learn. I'm also Dutch, so please feel free to tell me in a very direct manner which parts make no sense at all 😂 .

Related Issues

I believe this PR can contribute to resolving the below:

• [Filters] Add task-level filters to DAG View - Graph tab #53044
  closes: Reimplement #29226 for Airflow 3 UI | Filter tasks to Upstream and/or Downstream #53458

But there are more functionalities listed there, that I left out of scope for this PR.

Testing

I am unsure what the procedure is for testing in the UI. I spun up Breeze (love it btw, amazing tool) and clicked around there. Please advice on any further testing procedures or reading materials.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[jscheffl]

Thanks for adding the feature!

In order not to add complexity to the layout with another dropdown, could you add the lineage filter to the existing "Options" dropdown - that would be a bit more consistent for the UX in my view?

[OscarLigthart]

Thanks for your swift reply!

In order not to add complexity to the layout with another dropdown, could you add the lineage filter to the existing "Options" dropdown - that would be a bit more consistent for the UX in my view?

I considered it. The reason I didn't was because a user would not be able to see whether a DAG is filtered or not on first glance. With this extra dropdown there is an instant feedback showing back to the user. Curious to hear your thoughts on that approach.

That being said, more than happy to move it to options. For us the feature is the more important thing.

I have an additional question: I realized that I have only implemented it for the graph view, and deliberately left out the task overview. However, in Airflow 2 it propagated to both. Shall I try to implement it for both in this PR as well?

Example:

to:

[jscheffl]

I considered it. The reason I didn't was because a user would not be able to see whether a DAG is filtered or not on first glance. With this extra dropdown there is an instant feedback showing back to the user. Curious to hear your thoughts on that approach.

That being said, more than happy to move it to options. For us the feature is the more important thing.

Then before you rework and there might be different opinions, I think it is fair to also wait a second for other UI mainteainers feedback for UX - @pierrejeambrun / @bbovenzi What is your opinion about the UX?

[OscarLigthart]

Perfect, in the meantime I took the liberty of ensuring the behaviour is consistent between the Grid and the Graph 👍 . This also simplified the code a bit.

Looking forward to hearing from the other UX maintainers!

[r-richmond]

Hey Oscar thanks for tackling this p.s. #53458 was submitted for exactly this.

[OscarLigthart]

Hey Oscar thanks for tackling this p.s. #53458 was submitted for exactly this.

Ah I totally missed that one, thanks for the heads up! Regarding the other PR, let me know how you prefer to go ahead. Happy to continue with this PR, I have some time available for it.

[pierrejeambrun]

Super cool, I like it a lot.

I'm fine with having that as a separate dropdown than 'settings'.

I would have to play with it more but I think it's going in the right direction

[bbovenzi]

• I say we move forward with this PR. We haven't heard from the other PR author in a while.

• Let's definitely not call it Lineage. That's a different concept. Just "Filter" is ok.

• I think a separate dropdown is ok. I was tempted to put it in Options since that has plenty of other state that affects how the graph or grid are rendered, but it is important to make any filter very obvious.

[OscarLigthart]

Thanks all for the feedback. I will try to find some time today or tomorrow to implement and get back to you 👍

[OscarLigthart]

@bbovenzi I tried incorporating the feedback points you left. Thanks for the detailed review, points made a lot of sense to me. Please keep them coming!

@ashb, @bbovenzi , I changed the name to TaskStreamFilter. I wanted to be a bit more specific than just TaskFilter, as I think there are more variables that people can use for filtering tasks. However, I'm still not entirely sold by the name. So very open to suggestions.

[bbovenzi]

Nice work.

I think a lot of this PR can still be cleaned up.

We should make this filter the same as any table filter in Airflow and make use of searchParams. ?root=task_id&upstream=true. Then we can avoid all of this prop drilling and state management and any component that needs to be aware of the root can just check the searchParams.

[OscarLigthart]

We should make this filter the same as any table filter in Airflow and make use of searchParams. ?root=task_id&upstream=true. Then we can avoid all of this prop drilling and state management and any component that needs to be aware of the root can just check the searchParams.

That is actually an amazing piece of feedback! I should've thought of that before 😅 . Implemented!

Cleans up the hook and a lot of this state juggling.

[bbovenzi]

Just have some linting issue. Please run pnpm lint:fix and pnpm format locally

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.