Add back dag parsing pre-import optimization #50371

Lzzz666 · 2025-05-08T19:57:31Z

Related Issue

This PR reintroduces the pre-import optimization for DAG parsing originally added in Airflow 2.6.0 (#30495) but missing in Airflow 3.0. It reduces DAG parsing time by importing common modules in the parent process before forking.

I referred to #30495 and made modifications to 'processor.py' based on its implementation. I understand the issue is currently assigned to @kevinhongzl, and I’m happy to collaborate or adjust direction depending on their progress.

Looking forward to feedback!

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

airflow-core/src/airflow/dag_processing/processor.py

jason810496

Thanks for the PR, just double checked with #30495 and left some nits.

airflow-core/src/airflow/dag_processing/processor.py

jason810496

Nice! Thanks for the change.

airflow-core/src/airflow/dag_processing/processor.py

Lee-W

Also we might need to add a test for it

Lzzz666 · 2025-05-13T06:53:00Z

Also we might need to add a test for it

I’ll try adding a unit test for it.

Lzzz666 · 2025-05-15T10:25:33Z

Hi, I attempted to refactor the logic into a function.

This refactoring provides the following benefits:

Modularity: encapsulates the pre-import logic in a dedicated function
Easier testing: allows the _pre_import_airflow_modules function to be unit-tested in isolation
Improved readability: the function name clearly indicates its purpose

airflow-core/tests/unit/dag_processing/test_processor.py

jason810496

Thanks for the change.

Only a small nit on the test case naming: using test__pre_import_airflow_modules as a prefix would make it easier to identify the corresponding unit tests.

airflow-core/tests/unit/dag_processing/test_processor.py

Lee-W

Left a few nits, but overall looks good

airflow-core/src/airflow/dag_processing/processor.py

airflow-core/tests/unit/dag_processing/test_processor.py

Lee-W · 2025-05-19T08:06:35Z

@jedcunningham @ephraimbuddy not sure whether it's something missed or intentionally removed. But it looks good to me. I'm planning on merging it early tomorrow

airflow-core/src/airflow/dag_processing/processor.py

Lee-W · 2025-05-19T14:47:46Z

I just revoke my approval.

airflow-core/src/airflow/dag_processing/processor.py

amoghrajesh · 2025-06-04T06:49:15Z

@Lzzz666 would you be able to take a look at the review comments on this PR?

ephraimbuddy

LGTM

airflow-core/src/airflow/dag_processing/processor.py

airflow-core/tests/unit/dag_processing/test_processor.py

Lzzz666 · 2025-06-27T11:58:30Z

I tried to make an import error, then repaired the module, and the dag was retried and loaded successfully.

2025-06-27.7.48.40.mov

uranusjr · 2025-07-02T06:37:30Z

I’m going to merge this, let’s see what happens.

github-actions · 2025-07-02T07:10:55Z

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker d3bddfd v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

uranusjr · 2025-07-02T07:11:48Z

@Lzzz666 Can you help with backporting this manually?

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

Lzzz666 · 2025-07-04T04:29:27Z

Manual backport done in #52698

…0371) (#52698) * Add back dag parsing pre-import optimization (#50371) Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Apply suggestions from code review --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

Lzzz666 requested review from ephraimbuddy and jedcunningham as code owners May 8, 2025 19:57

boring-cyborg bot added the area:DAG-processing label May 8, 2025

Lee-W self-requested a review May 9, 2025 13:58

Lee-W reviewed May 9, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

jason810496 reviewed May 10, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

Lzzz666 marked this pull request as draft May 10, 2025 04:09

Lzzz666 marked this pull request as ready for review May 10, 2025 07:01

jason810496 reviewed May 11, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

jason810496 approved these changes May 12, 2025

View reviewed changes

Lee-W reviewed May 12, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

Lee-W reviewed May 12, 2025

View reviewed changes

eladkal added this to the Airflow 3.0.2 milestone May 12, 2025

eladkal added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label May 12, 2025

Lzzz666 marked this pull request as draft May 14, 2025 14:05

Lee-W reviewed May 15, 2025

View reviewed changes

airflow-core/tests/unit/dag_processing/test_processor.py Outdated Show resolved Hide resolved

Lzzz666 marked this pull request as ready for review May 15, 2025 15:07

jason810496 reviewed May 16, 2025

View reviewed changes

Lee-W approved these changes May 19, 2025

View reviewed changes

ephraimbuddy reviewed May 19, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

ephraimbuddy reviewed May 19, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

Lee-W self-requested a review May 19, 2025 14:47

eladkal reviewed May 22, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

vatsrahul1001 modified the milestones: Airflow 3.0.2, Airflow 3.0.3 Jun 3, 2025

ephraimbuddy reviewed Jun 25, 2025

View reviewed changes

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

airflow-core/src/airflow/dag_processing/processor.py Outdated Show resolved Hide resolved

ashb reviewed Jun 25, 2025

View reviewed changes

airflow-core/tests/unit/dag_processing/test_processor.py Outdated Show resolved Hide resolved

airflow-core/tests/unit/dag_processing/test_processor.py Outdated Show resolved Hide resolved

Refactor logger usage in DAG file processor tests

a39b16a

Lzzz666 requested a review from ashb June 27, 2025 02:02

Lzzz666 requested a review from eladkal June 27, 2025 11:58

Merge branch 'main' into main

2c6e23e

uranusjr merged commit d3bddfd into apache:main Jul 2, 2025
56 checks passed

Lzzz666 added a commit to Lzzz666/airflow that referenced this pull request Jul 2, 2025

Add back dag parsing pre-import optimization (apache#50371)

ea9ac58

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

Lzzz666 added a commit to Lzzz666/airflow that referenced this pull request Jul 2, 2025

Add back dag parsing pre-import optimization (apache#50371)

fb49b00

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

Lzzz666 mentioned this pull request Jul 2, 2025

[Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) #52698

Merged

kaxil pushed a commit to Lzzz666/airflow that referenced this pull request Jul 2, 2025

Add back dag parsing pre-import optimization (apache#50371)

7d1c7f4

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

kaxil pushed a commit to Lzzz666/airflow that referenced this pull request Jul 2, 2025

Add back dag parsing pre-import optimization (apache#50371)

f8270eb

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

kaxil modified the milestones: Airflow 3.0.3, Airflow 3.0.4 Jul 2, 2025

ashb mentioned this pull request Aug 5, 2025

Status of testing Apache Airflow 3.0.4rc2 and TaskSDK 1.0.4rc1 #54124

Closed

82 tasks

wjddn279 mentioned this pull request Dec 1, 2025

add pre loading of airflow module in dag-processor using runtime diff #58890

Closed

Add back dag parsing pre-import optimization #50371

Add back dag parsing pre-import optimization #50371

Uh oh!

Conversation

Lzzz666 commented May 8, 2025 • edited by eladkal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

Lzzz666 commented May 13, 2025

Uh oh!

Lzzz666 commented May 15, 2025

Uh oh!

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lee-W commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

Lee-W commented May 19, 2025

Uh oh!

Uh oh!

amoghrajesh commented Jun 4, 2025

Uh oh!

ephraimbuddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lzzz666 commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uranusjr commented Jul 2, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 2, 2025

Backport failed to create: v3-0-test. View the failure log Run details

Uh oh!

uranusjr commented Jul 2, 2025

Uh oh!

Lzzz666 commented Jul 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Lzzz666 commented May 8, 2025 •

edited by eladkal

Loading

Lzzz666 commented Jun 27, 2025 •

edited

Loading