AIP-84 Add multisort to dags list request #46383 #47440

prasad-madine · 2025-03-06T07:57:59Z

Description:

In the current UI, the DAGs list is sorted by default to display DAGs with the most recent DAG run first, achieved by sorting with -last_dag_run_start_date. However, this approach leads to an unintended consequence: DAGs that have never been run are displayed in reverse alphabetical order, which can be counterintuitive for users. Additionally, reversing the order of last_dag_run_start_date also affects the alphabetical order of DAGs.

backend part of: #46383

Add support for multi sort in the API

Test cases screenshot

boring-cyborg · 2025-03-06T07:58:02Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

pierrejeambrun

Nice, thanks for the pull request 🎉

I think we need:

To allow multi sorting on the get dags endpoint to solve the related issue.
Add tests for the get dags endpoint with multiple sort in the query and assert that it is doing what we expect.

airflow/api_fastapi/common/parameters.py

pierrejeambrun

Nice, we need some tests too.

airflow/api_fastapi/common/parameters.py

airflow/api_fastapi/core_api/routes/public/dags.py

pierrejeambrun · 2025-03-07T12:23:00Z

Don't hesitate to resolve comments that you have addressed. This will make the review easier, thanks.

prasad-madine · 2025-03-07T12:26:50Z

Nice, we need some tests too.

Yes, currently working on that, will update the PR once it is done.
Thankyou

prasad-madine · 2025-03-07T12:27:25Z

Don't hesitate to resolve comments that you have addressed. This will make the review easier, thanks.

I thought to do it after writing test cases.

prasad-madine · 2025-03-13T07:04:26Z

Hi @pierrejeambrun , I have added few tests, COuld you please review it and let me know if any additional things to be required.
Thankyou

…bug-add-multisort-to-dags-list-request-apache#46383

pierrejeambrun

Thanks for the PR. Looking good overall. One couple of comments only.

CI need fixing (tests, static check etc.), can you also rebase the branch solve conflicts.

airflow/api_fastapi/common/parameters.py

tests/api_fastapi/core_api/routes/public/test_dags.py

…bug-add-multisort-to-dags-list-request-apache#46383

…e#46383

prasad-madine · 2025-03-27T04:57:32Z

@pierrejeambrun could you please review the PR, If any changes needs to be done, I'll address those.
Thankyou

pierrejeambrun

Thanks.

A few comments. Also the CI needs to be fixed:

You can install pre-commits to run static checks locally. https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst
You can start test locally with a local virtualenv and pytest or with breeze. All of that is described in the contribution docs (https://github.com/apache/airflow/tree/main/contributing-docs, https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst
https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst)

pierrejeambrun · 2025-03-31T08:24:55Z

airflow-core/src/airflow/api_fastapi/common/parameters.py

        self.to_replace = to_replace

    def to_orm(self, select: Select) -> Select:
+        MAX_SORT_PARAMS = 5


I think the limit should be made greater.

We already have table with more than 5 columns and the UI could reach that pretty soon.

Going for 10 or 20 shouldn't hurt. (all our different db should be able to handle much more anyway)

pierrejeambrun · 2025-03-31T08:31:29Z

airflow-core/src/airflow/api_fastapi/common/parameters.py

+        def inner(order_by: list[str] | str | None = Query(default)) -> SortParam:
+            if order_by is None:
+                order_by = [self.get_primary_key_string()]
+            elif isinstance(order_by, str):
+                order_by = [order_by]
+            return self.set_value(order_by)


I'm not sure it's exactly equivalent. If there is no order_by query param specified, we shouldn't add any by default and let the query default order operate:

order_by = None => don't update the query at all and let the default ordering of the query operate.

order_by = [] or [""] -> fill with [primary_key_string] (or raise validation error)

pierrejeambrun · 2025-03-31T08:34:51Z

airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dags.py

+            ({"order_by": ["last_run_state", "dag_display_name"], "only_active": False}, 3, [DAG1_ID, DAG3_ID, DAG2_ID]),
+            ({"order_by": ["-last_run_state", "dag_display_name"], "only_active": False}, 3, [DAG3_ID, DAG1_ID, DAG2_ID]),
+            ({"order_by": ["last_run_start_date", "dag_display_name"], "only_active": False}, 3, [DAG1_ID, DAG3_ID, DAG2_ID]),
+            ({"order_by": ["-last_run_start_date", "dag_display_name"], "only_active": False}, 3, [DAG3_ID, DAG1_ID, DAG2_ID]),
+            ({"order_by": ["-dag_id", "dag_display_name"]}, 2, [DAG2_ID, DAG1_ID]),
+            ({"order_by": ["dag_id", "dag_display_name"]}, 2, [DAG1_ID, DAG2_ID]),
+            ({"order_by": ["dag_display_name", "dag_id"]}, 2, [DAG1_ID, DAG2_ID]),
+            ({"order_by": ["last_run_state"], "only_active": False}, 3, [DAG1_ID, DAG3_ID, DAG2_ID]),
+            ({"order_by": ["-last_run_state"], "only_active": False}, 3, [DAG3_ID, DAG1_ID, DAG2_ID]),
+            ({"order_by": ["last_run_start_date"], "only_active": False}, 3, [DAG1_ID, DAG3_ID, DAG2_ID]),
+            ({"order_by": ["-last_run_start_date"], "only_active": False}, 3, [DAG3_ID, DAG1_ID, DAG2_ID])
+
+            ({"order_by": ["dag_display_name", "-next_dagrun"]}, 2, [DAG1_ID, DAG2_ID]),
+            ({"order_by": ["last_run_state", "-dag_display_name", "dag_id"]}, 2, [DAG1_ID, DAG2_ID]),
+            ({"order_by": ["-last_run_start_date", "dag_display_name", "next_dagrun", "dag_id"]}, 2, [ DAG1_ID, DAG2_ID]),
+            ({"order_by": ["dag_display_name", "-last_run_state", "next_dagrun", "dag_id", "last_run_start_date"]}, 2, [DAG1_ID, DAG2_ID]),


We still need two tests with:

({"order_by": ["criteria_1", "criteria_2"]}, 2, [id1, id2, id3]), ({"order_by": ["criteria_1", "-criteria_2"]}, 2, [different sequence of id, proving that criteria_2 is being considerd]),

pierrejeambrun · 2025-03-31T08:35:51Z

airflow-core/src/airflow/api_fastapi/common/parameters.py

+            elif isinstance(order_by, str):
+                order_by = [order_by]


This shouldn't be possible. FastAPI validation will prevent such wrong input and will do what's necessary to either get a list[str] or fail with a 422 validation error if not possible.

pierrejeambrun · 2025-03-31T08:39:55Z

airflow-core/src/airflow/api_fastapi/common/parameters.py

-        def inner(order_by: str = default or self.get_primary_key_string()) -> SortParam:
-            return self.set_value(self.get_primary_key_string() if order_by == "" else order_by)
+    def dynamic_depends(self, default: list[str] | None = None) -> Callable:
+        def inner(order_by: list[str] | str | None = Query(default)) -> SortParam:


Suggested change

def inner(order_by: list[str] | str | None = Query(default)) -> SortParam:

def inner(order_by: list[str] | None = Query(default)) -> SortParam:

pierrejeambrun · 2025-04-09T09:40:50Z

@prasad-madine are you able to move forward, or do you need help ?

prasad-madine · 2025-04-09T10:09:55Z

@prasad-madine are you able to move forward, or do you need help ?

@pierrejeambrun I've made changes, but facing issues in setup with latest code, I need to test it. I've also tried to setup application again. Need some help.

…e#46383

pierrejeambrun · 2025-04-14T09:04:22Z

Make sure your images have been rebuilt to be up to date. (When you start breeze you should see a prompt asking if you want to rebuild or not, you need to answer yes to both of the prompt).

You can try in incognito mode in your browser. You might have some old cached version of the localstorage or cookies creating some problem.

Otherwise you can stop breeze then run breeze down and start breeze again.

If the issue persists you can try breeze cleanup.

prasad-madine · 2025-04-14T09:08:56Z

Make sure your images have been rebuilt to be up to date. (When you start breeze you should see a prompt asking if you want to rebuild or not, you need to answer yes to both of the prompt).

You can try in incognito mode in your browser. You might have some old cached version of the localstorage or cookies creating some problem.

Otherwise you can stop breeze then run breeze down and start breeze again.

If the issue persists you can try breeze cleanup.

I've also tried incognito and also tried removing networks, volumes and docker images and rebuilt the application again, but still facing the same issue.

…e#46383

pierrejeambrun · 2025-04-14T09:33:15Z

Something must be wrong, you can try on main. Be sure to rebuild images.

It looks like the connection is refused, the api-server is not reached. Also you can try with --dev-mode or without this flag.

pierrejeambrun · 2025-04-14T09:34:13Z

That might be wrong on the auth manager asset compilation side for --dev-mode

pierrejeambrun · 2025-04-16T13:11:58Z

@prasad-madine did you manage to solve your local setup issue ?

pierrejeambrun · 2025-05-15T14:41:26Z

@prasad-madine Any progress on that?

prasad-madine · 2025-05-15T14:56:29Z

No @pierrejeambrun , still the issue with application setup is not resolved.

pierrejeambrun · 2025-05-16T10:18:45Z

This does not happen on latest main. You can rebuild the images and start breeze again, you should be able to access the UI. Breeze documentation can be found here https://github.com/apache/airflow/tree/main/dev/breeze/doc.

You can run a breeze down then breeze cleanup to start over. That should definitely work on main branch. Also check that you don't have anything interfering in your files folder (env variables, entry point script etc...), default setup should work.

github-actions · 2025-07-01T00:21:12Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

pierrejeambrun · 2025-07-09T14:06:13Z

I am working on this, and will open a separate PR when it's ready

pierrejeambrun · 2025-07-16T09:06:28Z

Closing in favor of #53408

fix:bug Add multisort to dags list request apache#46383

59eab52

prasad-madine requested review from ephraimbuddy and pierrejeambrun as code owners March 6, 2025 07:57

boring-cyborg bot added the area:API Airflow's REST/HTTP API label Mar 6, 2025

prasad-madine mentioned this pull request Mar 6, 2025

AIP-84 | Add multisort to dags list request #46383

Open

pierrejeambrun changed the title ~~fix:bug Add multisort to dags list request #46383~~ AIP-84 Add multisort to dags list request #46383 Mar 6, 2025

pierrejeambrun added the AIP-84 Modern Rest API label Mar 6, 2025

pierrejeambrun added this to AIP-84 MODERN REST API Mar 6, 2025

pierrejeambrun moved this to In Progress in AIP-84 MODERN REST API Mar 6, 2025

pierrejeambrun added this to the Airflow 3.0.0 milestone Mar 6, 2025

pierrejeambrun reviewed Mar 6, 2025

View reviewed changes

prasad-madine added 2 commits March 7, 2025 10:24

fix:review comments

a03fb8a

fix:review comments

96085fb

pierrejeambrun reviewed Mar 7, 2025

View reviewed changes

airflow/api_fastapi/common/parameters.py Outdated Show resolved Hide resolved

airflow/api_fastapi/core_api/routes/public/dags.py Outdated Show resolved Hide resolved

fix: Added test cases and fixed logic

05e5104

Merge branch 'main' of https://github.com/prasad-madine/airflow into …

43d2b5a

…bug-add-multisort-to-dags-list-request-apache#46383

prasad-madine requested a review from pierrejeambrun March 14, 2025 04:48

mariana-marcal-santana mentioned this pull request Mar 17, 2025

Implement multisort in DAGs table #47849

Closed

pierrejeambrun reviewed Mar 17, 2025

View reviewed changes

airflow/api_fastapi/common/parameters.py Outdated Show resolved Hide resolved

tests/api_fastapi/core_api/routes/public/test_dags.py Outdated Show resolved Hide resolved

prasad-madine added 2 commits March 26, 2025 17:57

fix: addressed the comments

d1a769c

Merge branch 'main' of https://github.com/prasad-madine/airflow into …

f21e730

…bug-add-multisort-to-dags-list-request-apache#46383

prasad-madine requested a review from rawwar as a code owner March 26, 2025 12:28

prasad-madine requested a review from pierrejeambrun March 26, 2025 12:32

prasad-madine added 2 commits March 26, 2025 20:25

Merge branch 'main' into bug-add-multisort-to-dags-list-request-apach…

61997f1

…e#46383

Merge branch 'main' into bug-add-multisort-to-dags-list-request-apach…

2944655

…e#46383

pierrejeambrun reviewed Mar 31, 2025

View reviewed changes

prasad-madine added 2 commits April 9, 2025 15:49

Merge branch 'main' into bug-add-multisort-to-dags-list-request-apach…

358c5ff

…e#46383

Merge branch 'main' into bug-add-multisort-to-dags-list-request-apach…

3065aad

…e#46383

Merge branch 'main' into bug-add-multisort-to-dags-list-request-apach…

6098b2f

…e#46383

prasad-madine requested a review from jason810496 as a code owner April 14, 2025 09:09

kaxil modified the milestones: Airflow 3.0.0, Airflow 3.1+ Apr 15, 2025

rawwar assigned prasad-madine Apr 19, 2025

eladkal modified the milestones: Airflow 3.1+, Airflow 3.1.0 Apr 21, 2025

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 1, 2025

pierrejeambrun closed this Jul 16, 2025

kaxil removed this from the Airflow 3.1.0 milestone Sep 14, 2025

	def inner(order_by: list[str] \| str \| None = Query(default)) -> SortParam:
	def inner(order_by: list[str] \| None = Query(default)) -> SortParam:

AIP-84 Add multisort to dags list request #46383 #47440

AIP-84 Add multisort to dags list request #46383 #47440

Uh oh!

Conversation

prasad-madine commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Test cases screenshot

Uh oh!

boring-cyborg bot commented Mar 6, 2025

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun commented Mar 7, 2025

Uh oh!

prasad-madine commented Mar 7, 2025

Uh oh!

prasad-madine commented Mar 7, 2025

Uh oh!

prasad-madine commented Mar 13, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

prasad-madine commented Mar 27, 2025

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun commented Apr 9, 2025

Uh oh!

prasad-madine commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pierrejeambrun commented Apr 14, 2025

Uh oh!

prasad-madine commented Apr 14, 2025

Uh oh!

pierrejeambrun commented Apr 14, 2025

Uh oh!

pierrejeambrun commented Apr 14, 2025

Uh oh!

pierrejeambrun commented Apr 16, 2025

Uh oh!

pierrejeambrun commented May 15, 2025

Uh oh!

prasad-madine commented May 15, 2025

prasad-madine commented Mar 6, 2025 •

edited

Loading

pierrejeambrun left a comment •

edited

Loading

pierrejeambrun left a comment •

edited

Loading

pierrejeambrun Mar 31, 2025 •

edited

Loading

prasad-madine commented Apr 9, 2025 •

edited

Loading

pierrejeambrun commented May 16, 2025 •

edited

Loading