feat(api/ui)#50779 : add asset group filtering and asset group views #51682

davidfgcorreia · 2025-06-13T09:53:05Z

Add backend and OpenAPI support for filtering assets by group (group_pattern) in /assets endpoint.

Implement multi-asset dependency queries in /ui/dependencies endpoint with node_ids parameter.

Add new frontend pages and components for asset group views, including group graph and sidebar.

Refactor asset list and group list to support group filtering, merging, and deduplication.

Add and update tests for group filtering, group views, and multi-asset dependency queries.

introduced small bug in caching on the asset_grath.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

bugraoz93

API side looks good! Could you please fix static checks? It is failing for Compile / format / lint UI

davidfgcorreia · 2025-06-13T18:48:20Z

Runned precommit locally and got zero errors how can I replicate this setup and make sure it passes consistently?

guan404ming

Thanks, this feature is great. Feel free to ask anything.

guan404ming · 2025-06-19T20:47:17Z

airflow-core/src/airflow/ui/src/pages/Asset/AssetGroupSidebar.tsx

+        <HStack gap={4} mb={2}>
+          <Box>
+            <Text color="chakra-fg" fontSize="xs">
+              {/* i18n: Producing Tasks label */}Producing Tasks


We currently disable literal string in ui due to i18n thus you need to update the translation file like en/assets.json or en/common.json and use useTranslation hook to access them. You could also run npm run lint in ui folder to see your lint error.

davidfgcorreia · 2025-06-21T09:03:20Z

Thanks, this feature is great. Feel free to ask anything.

Yes, I have a question. I created a new file called sidebar.tsx, but I'm not able to catch the errors in it , even though the runner here on Git is marking them. Do you know what might be going on?

pierrejeambrun

Nice effort!

I believe there are a few things to update before we can merge but this can be a great feature and addition for Airflow 3.1.0

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dependencies.py

airflow-core/src/airflow/ui/src/pages/AssetsGroupedList/AssetsGroupedList.tsx

airflow-core/src/airflow/ui/src/pages/AssetsList/AssetsList.tsx

airflow-core/src/airflow/ui/src/queries/useMultiDependencyGraph.ts

pierrejeambrun · 2025-06-23T11:01:26Z

Yes, I have a question. I created a new file called sidebar.tsx, but I'm not able to catch the errors in it , even though the runner here on Git is marking them. Do you know what might be going on?

Be sure to be rebased on latest main with latest devtools updates (We recently merged a few improvement to this part). You can then run pre-commit hooks or eslint:

pre-commit run ts-compile-lint-ui (with your files stages) or pre-commit run ts-compile-lint-ui --all-files (for all files) This is what is run in the CI so you should be able to reproduce
or pnpm eslint --fix your_file (maybe don't run that, I'm having trouble running it locally at the moment, I thing something might be broken in the setup)

davidfgcorreia · 2025-06-26T14:19:07Z

I'm getting a mypy error when running the pre-commit hook for airflow-core. Here's the stack trace:

AssertionError: Should never get here in normal mode, got TypeAlias:numpy.bool_ instead of TypeInfo
It suggests running breeze ci-image build --python 3.9 or breeze down --cleanup-mypy-cache, but I'm not sure what's actually causing this.
Any idea what might be going on or how to fix it?

pierrejeambrun

No idea what is wrong in your setup. You can run pre-commit on main to be sure that everything is in order. It is really hard for me to help you without more context and iterating. Everything should be detailed in the contribution docs where you can find all the steps and troubleshooting help.

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dependencies.py

davidfgcorreia · 2025-06-29T16:03:01Z

Hi, the pipeline is currently failing on Tests AMD / CI image checks / Static checks (pull_request), specifically at the step "Generate SVG from Airflow CTL Commands".

However, when I run the checks locally:

pre-commit run (on modified files only) skips this check, as expected, since I haven't modified any related files;

pre-commit run --all-files passes the check successfully, including the "Generate SVG from Airflow CTL Commands" step.

Please let me know if there's anything else I should look into or adjust. Thanks!

potiuk · 2025-06-29T16:13:09Z

Hi, the pipeline is currently failing on Tests AMD / CI image checks / Static checks (pull_request), specifically at the step "Generate SVG from Airflow CTL Commands".

However, when I run the checks locally:

pre-commit run (on modified files only) skips this check, as expected, since I haven't modified any related files;

pre-commit run --all-files passes the check successfully, including the "Generate SVG from Airflow CTL Commands" step.

Please let me know if there's anything else I should look into or adjust. Thanks!

That was temporary. As usual - rebase. Always Be Rebased. There is another issue in main being fixed as we speak though so you might wait a few minutes.

potiuk · 2025-06-29T16:14:28Z

This is a PR that targets fixing the current main "generate constraints" failure #52464 - once merged, rebasing your PR should fix unrelated issue.

…views Add backend and OpenAPI support for filtering assets by group (`group_pattern`) in /assets endpoint. Implement multi-asset dependency queries in /ui/dependencies endpoint with node_ids parameter. Add new frontend pages and components for asset group views, including group graph and sidebar. Refactor asset list and group list to support group filtering, merging, and deduplication. Add and update tests for group filtering, group views, and multi-asset dependency queries. Co-authored-by: Francisco Núncio <francisco.nuncio@tecnico.ulisboa.pt>

- Added `/assets/groups` endpoint to return asset groups with their assets and counts. - Added `AssetGroupResponse` and `AssetGroupCollectionResponse` models to API and OpenAPI schemas. - Updated backend, OpenAPI YAML, and TypeScript types to support asset group listing and filtering. - Updated frontend to use new `useAssetServiceGetAssetGroups` hook for grouped asset views and sidebar. - Refactored asset list and group pages to use new group API and improved i18n. - Updated dependencies API and frontend to use exploded `node_ids` query parameter (array, not comma-separated). - Updated backend logic for `/dependencies` to return 404 if any requested node_id is missing. - Updated and fixed unit tests for dependencies and asset groups to match new API and error handling. - Improved search and filtering for assets and asset groups in UI.

potiuk · 2025-06-29T16:18:53Z

Rebased it for you after merging my PR.

davidfgcorreia · 2025-07-06T11:54:11Z

After my last changes, the scheduler query count tests are failing with a much higher number of queries than expected, even though I did not touch any scheduler or ORM/database logic. I double-checked and only changed API and UI code. Has anyone seen this before or have any idea what could cause this?

pierrejeambrun · 2025-07-10T11:40:40Z

@davidfgcorreia that could be transient, unrelated, you can close and reopen the PR to retrigger the CI, or just rebase or just push an amended commit.

pierrejeambrun

Thanks for addressing the previous comments.

I think the PR needs refinement, refactoring, simplifications before we can merge this, I think it still remains an important effort there.

That would be a great addition to 3.1.0

pierrejeambrun · 2025-08-08T13:53:59Z

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dependencies.py

+    node_id: str | None = None,
+    node_ids: list[str] = Query(None, description="List of node ids"),
+) -> BaseGraphResponse:
+    """Dependencies graph. Supports a single node_id or multiple node_ids as exploded query parameters (node_ids=foo&node_ids=bar)."""


Suggested change

"""Dependencies graph. Supports a single node_id or multiple node_ids as exploded query parameters (node_ids=foo&node_ids=bar)."""

"""Dependencies graph. Supports a single node_id or multiple node_ids

pierrejeambrun · 2025-08-08T13:54:43Z

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dependencies.py

-def get_dependencies(session: SessionDep, node_id: str | None = None) -> BaseGraphResponse:
-    """Dependencies graph."""
+def get_dependencies(
+    node_id: str | None = None,


Suggested change

node_id: str | None = None,

We should probably only use node_ids even for single node to simplify the API. That's private API so it doens't have to be backward compatible.

pierrejeambrun · 2025-08-08T13:56:59Z

airflow-core/src/airflow/ui/src/pages/AssetsGroupedList/AssetsGroupedList.tsx

+const truncate = (str: string, max = 32) => (str.length > max ? `${str.slice(0, max - 3)}...` : str);
+
+const nameCellRenderer = ({ row: { original } }: { row: { original: AssetResponse } }) => (
+  <NameCell original={original} />
+);
+const lastAssetEventCellRenderer = ({ row: { original } }: { row: { original: AssetResponse } }) => (
+  <LastAssetEventCell original={original} />
+);
+const groupCellRenderer = ({ row: { original } }: { row: { original: AssetResponse } }) => (
+  <GroupCell original={original} />
+);
+const consumingDagsCellRenderer = ({ row: { original } }: { row: { original: AssetResponse } }) => (
+  <ConsumingDagsCell original={original} />
+);
+const producingTasksCellRenderer = ({ row: { original } }: { row: { original: AssetResponse } }) => (
+  <ProducingTasksCell original={original} />
+);
+const triggerCellRenderer = ({ row: { original } }: { row: { original: AssetResponse } }) => (
+  <TriggerCell original={original} />
+);
+


Similarly to other columns definition, inline this directly in the columns definition.

pierrejeambrun · 2025-08-08T13:57:27Z

airflow-core/src/airflow/ui/src/pages/AssetsGroupedList/AssetsGroupedList.tsx

+  const [sort] = sorting;
+  const orderBy = sort ? `${sort.desc ? "-" : ""}${sort.id}` : undefined;
+
+  // Use o novo hook para buscar grupos paginados


Comment not in english.

pierrejeambrun · 2025-08-08T13:59:02Z

airflow-core/src/airflow/ui/src/pages/AssetsList/AssetsList.tsx

  const [sort] = sorting;
  const orderBy = sort ? `${sort.desc ? "-" : ""}${sort.id}` : undefined;

+  const searchMatchMode = "any"; // "any" para OR, "all" para AND


comment not in english.

pierrejeambrun · 2025-08-08T14:26:56Z

airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py

+    assets_query = (
+        select(AssetModel)
+        .where(AssetModel.group.in_(groups))
+        .options(
+            subqueryload(AssetModel.scheduled_dags),
+            subqueryload(AssetModel.producing_tasks),
+            subqueryload(AssetModel.consuming_tasks),
+        )
+    )
+    assets = session.scalars(assets_query).all()


This will query all assets from the db. Can't happen I think

pierrejeambrun · 2025-08-08T14:27:22Z

airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py

+    group_query = (
+        select(AssetModel.group, func.count(AssetModel.id))
+        .where(
+            AssetModel.active.has() if only_active.value else True,
+            *([AssetModel.group == group] if group else []),
+        )
+        .group_by(AssetModel.group)
+    )


This will query all assets from the db grouped by asset group, but that could be thousands of them, that' not scalable.

pierrejeambrun · 2025-08-08T14:27:42Z

airflow-core/src/airflow/api_fastapi/core_api/services/ui/dependencies.py

+    if len(node_ids) == 1:
+        node_id = node_ids[0]
+        filtered_connected_components = [cc for cc in connected_components if node_id in cc]
+        if len(filtered_connected_components) != 1:
+            raise ValueError(
+                f"Unique connected component not found, got {filtered_connected_components} for connected components of node {node_id}, expected only 1 connected component."
+            )
+        connected_component = filtered_connected_components[0]
+        filtered_nodes = [node for node in nodes if node["id"] in connected_component]
+        filtered_edges = [
+            edge
+            for edge in edges
+            if (edge["source_id"] in connected_component and edge["target_id"] in connected_component)
+        ]
+        return {"nodes": filtered_nodes, "edges": filtered_edges}
+    # Multiple node_ids: merge all relevant components
+    relevant_components = [cc for cc in connected_components if any(nid in cc for nid in node_ids)]
+    # NEW: Check that all node_ids are present in the merged set
+    if not relevant_components:
+        raise ValueError(f"No connected component found for node(s) {node_ids}.")
+    merged_node_ids = set().union(*relevant_components)
+    missing = [nid for nid in node_ids if nid not in merged_node_ids]
+    if missing:
+        raise ValueError(f"No connected component found for node(s) {missing}.")
+    filtered_nodes = [node for node in nodes if node["id"] in merged_node_ids]
+    filtered_edges = [
        edge
        for edge in edges
-        if (edge["source_id"] in connected_component and edge["target_id"] in connected_component)
+        if (edge["source_id"] in merged_node_ids and edge["target_id"] in merged_node_ids)


Needs refactoring, that's really hard to read .

pierrejeambrun · 2025-08-08T14:28:05Z

airflow-core/src/airflow/ui/src/pages/Asset/AssetGroupSidebar.tsx

+          <b>{translate("group", { ns: "assets" })}</b>
+          <br />


use chakra.p, chakra.br, etc.

pierrejeambrun · 2025-08-08T14:28:45Z

airflow-core/src/airflow/ui/src/pages/Asset/AssetGroupSidebar.tsx

+              <Text color="chakra-fg-subtle" fontSize="xs">
+                {ev.timestamp}
+              </Text>


We probably shouldn't display timestamp as text, check other location of the app for Time component.

jason810496 · 2025-08-09T17:35:45Z

Hi @davidfgcorreia, thanks for the contribution!

May I ask if this PR was closed by accident?

davidfgcorreia · 2025-08-10T19:50:22Z

I can't work on this issue no longer.
Sorry for the trouble.

davidfgcorreia requested review from bbovenzi, bugraoz93, ephraimbuddy, jason810496, jscheffl, pierrejeambrun, rawwar, ryanahamilton and shubhamraj-git as code owners June 13, 2025 09:53

boring-cyborg bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Jun 13, 2025

bugraoz93 reviewed Jun 13, 2025

View reviewed changes

guan404ming reviewed Jun 19, 2025

View reviewed changes

pierrejeambrun reviewed Jun 23, 2025

View reviewed changes

pierrejeambrun reviewed Jun 27, 2025

View reviewed changes

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/dependencies.py Outdated Show resolved Hide resolved

davidfgcorreia force-pushed the feature_#50779 branch from 47ce735 to f71a6f5 Compare June 29, 2025 13:54

davidfgcorreia requested review from kaxil and potiuk as code owners June 29, 2025 13:54

davidfgcorreia force-pushed the feature_#50779 branch from f71a6f5 to 7bd5da7 Compare June 29, 2025 14:07

davidfgcorreia and others added 2 commits June 29, 2025 18:18

potiuk force-pushed the feature_#50779 branch from 7bd5da7 to 375f067 Compare June 29, 2025 16:18

pierrejeambrun closed this Aug 8, 2025

pierrejeambrun reopened this Aug 8, 2025

pierrejeambrun added this to the Airflow 3.1.0 milestone Aug 8, 2025

pierrejeambrun reviewed Aug 8, 2025

View reviewed changes

davidfgcorreia closed this Aug 9, 2025

pierrejeambrun mentioned this pull request Aug 21, 2025

Add tab for asset groups and add allow filtering for asset tags and URIs #50779

Open

2 tasks

kaxil removed this from the Airflow 3.1.0 milestone Sep 14, 2025

	"""Dependencies graph. Supports a single node_id or multiple node_ids as exploded query parameters (node_ids=foo&node_ids=bar)."""
	"""Dependencies graph. Supports a single node_id or multiple node_ids

feat(api/ui)#50779 : add asset group filtering and asset group views #51682

feat(api/ui)#50779 : add asset group filtering and asset group views #51682

Uh oh!

Conversation

davidfgcorreia commented Jun 13, 2025

Uh oh!

bugraoz93 left a comment

Choose a reason for hiding this comment

Uh oh!

davidfgcorreia commented Jun 13, 2025

Uh oh!

guan404ming left a comment

Choose a reason for hiding this comment

Uh oh!

guan404ming Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidfgcorreia commented Jun 21, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidfgcorreia commented Jun 26, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidfgcorreia commented Jun 29, 2025

Uh oh!

potiuk commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Jun 29, 2025

Uh oh!

potiuk commented Jun 29, 2025

Uh oh!

davidfgcorreia commented Jul 6, 2025

Uh oh!

pierrejeambrun commented Jul 10, 2025

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jason810496 commented Aug 9, 2025

Uh oh!

davidfgcorreia commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

guan404ming Jun 19, 2025 •

edited

Loading

pierrejeambrun commented Jun 23, 2025 •

edited

Loading

potiuk commented Jun 29, 2025 •

edited

Loading

davidfgcorreia commented Aug 10, 2025 •

edited

Loading