Fix n+1 query to fetch tags in the dags list page #57270
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `/public/dags` and `/ui/dags` endpoints were triggering n+1 queries when loading DAG tags - one query to fetch DAGs, then one additional query per DAG to fetch its tags. For deployments with many DAGs, this could cause significant performance degradation. Added `selectinload(DagModel.tags)` to `generate_dag_with_latest_run_query()` to eagerly load all tags in a single additional query instead of N separate queries. This reduces the total queries from O(N) to O(1) with respect to the number of DAGs. Example impact: - Before: 1 query for DAGs + 100 queries for tags (100 DAGs) = 101 queries - After: 1 query for DAGs + 1 query for all tags = 2 queries Added regression tests that verify query count doesn't scale linearly with the number of DAGs by comparing counts before and after adding more DAGs. Fixes apache#57241
kaxil
requested review from
ephraimbuddy,
jason810496,
pierrejeambrun and
rawwar
as code owners
jason810496
approved these changes
Oct 26, 2025
Member
jason810496
left a comment
jason810496
left a comment
There was a problem hiding this comment.
Thanks for the improvement!
Member
There was a problem hiding this comment.
Super cool, thanks.
That could help solve #56635, where loading dags were reported slow.
Or we might have another similar issue specific to ui/dags when fetching 'recent dag runs'.
pierrejeambrun
pushed a commit
to astronomer/airflow
that referenced
this pull request
Oct 30, 2025
(cherry picked from commit 5a28c44)
Member
kaxil
added a commit
that referenced
this pull request
Oct 30, 2025
17 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
/public/dagsand/ui/dagsendpoints were triggering n+1 queries when loading DAG tags - one query to fetch DAGs, then one additional query per DAG to fetch its tags. For deployments with many DAGs, this could cause significant performance degradation.Added
selectinload(DagModel.tags)togenerate_dag_with_latest_run_query()to eagerly load all tags in a single additional query instead of N separate queries. This reduces the total queries from O(N) to O(1) with respect to the number of DAGs.Example impact:
Added regression tests that verify query count doesn't scale linearly with the number of DAGs by comparing counts before and after adding more DAGs.
Fixes #57241
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.