Dags list/list-import-errors cli commands broken and don't track local DagBag

[dheerajturaga]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When developing dags, airflow dags list and airflow dags list-import-errors return No data found when the database has just been initialized. Also any update to the dags aren't reflected with these commands unless you run airflow dags reserialize. THe command only reads from db, it should also track localdag bag similar to airflow 2. This is a serious problem as these commands are used widely in dag development.

This is happening because these commands are reading from the db rather than tracking the local dagbag.
DagBag(read_dags_from_db=True)

What you think should happen instead?

Given we are always reading from the db in the case of list and list-import-errors we need to run reserialize before fetching these values from the database

How to reproduce

Seed a basic dag in the dags directory and run. Just use the defaults.

airflow db migrate
airflow dags list

Operating System

Ubuntu 24.04.1 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[bugraoz93]

I think this should already be from db. Because let's say you have airflow.cfg with proper jdbc connection and network rules, you can execute this command from a remote machine rather than in the component container or operating system then you won't have local DAGBag anymore. I don’t think this is a bug though
Reviewed the PR :)

[dheerajturaga]

@bugraoz93 Thats only if you have a dag-processor running. in airflow 2.10.5 this command reads directly from the repo rather than the db. The current implementation breaks local DagBag development and requires you to have a dag-processor job actively running during dag development. We should support both scenarios

I have a setup where I just have the following settings in a source_me in my dags repository and I use airflow dags list-import-errors
while editing my dags

AIRFLOW_HOME = '<dags_repo>/run'
AIRFLOW__CORE__DAGS_FOLDER = <dags_repo>
AIRFLOW__CORE__LOAD_EXAMPLES = false

in this setup airflow info will auto create a airflow.cfg and airflow.db in the dags_repo/run area. Its a downgrade if we need to airflow dags reserialize and then airflow dags list-import-errors every time instead of just airflow dags list-import-errors to figure out whats wrong ( which was the case in airflow 2 )

[bugraoz93]

@bugraoz93 Thats only if you have a dag-processor running. in airflow 2.10.5 this command reads directly from the repo rather than the db. The current implementation breaks local DagBag development and requires you to have a dag-processor job actively running during dag development. We should support both scenarios

I have a setup where I just have the following settings in a source_me in my dags repository and I use airflow dags list-import-errors
while editing my dags

AIRFLOW_HOME = '<dags_repo>/run'
AIRFLOW__CORE__DAGS_FOLDER = <dags_repo>
AIRFLOW__CORE__LOAD_EXAMPLES = false

in this setup airflow info will auto create a airflow.cfg and airflow.db in the dags_repo/run area. Its a downgrade if we need to airflow dags reserialize and then airflow dags list-import-errors every time instead of just airflow dags list-import-errors to figure out whats wrong ( which was the case in airflow 2 )

I want to clarify some parts. I see that the issue opened as 3.0.0, which is the current main branch. In Airflow 3.0 (main branch), there will be Dag Processors running. All the CLI commands are updated according to the dag bundle and versioning, which is managed by Dag Processors. So, in the context of 3.0.0, I think this is not a problem but already planned changes.
See https://github.com/apache/airflow/blob/main/airflow-core/newsfragments/45729.significant.rst

Current latest development version of 2.10
https://github.com/apache/airflow/tree/v2-10-test

Stable
https://github.com/apache/airflow/tree/v2-10-stable

[dheerajturaga]

@bugraoz93 , I have been testing airflow 3.0.0rc2 for a couple of days when I noticed this. So you are saying the only way to reflect the true state of the dags in a "out of the box/local" setting is to spin up a dag-processor in that local setup and then run the cli commands?

Don't you think we should add a --local arg ( or something similar ) to the list and list-import-errors to be able to iterate on during dag development in user's local areas?

[bugraoz93]

@bugraoz93 , I have been testing airflow 3.0.0rc2 for a couple of days when I noticed this. So you are saying the only way to reflect the true state of the dags in a "out of the box/local" setting is to spin up a dag-processor in that local setup and then run the cli commands?

Don't you think we should add a --local arg ( or something similar ) to the list and list-import-errors to be able to iterate on during dag development in user's local areas?

Yes, exactly for Airflow 3.0+. I don’t think that should be needed. How are you testing locally? If you are using Breeze, the start-airflow command should already run the standalone Dag Processors. Or if you are using docker-compose or helm charts in minikube or so. It should be already running out of the box.

Breeze
https://github.com/apache/airflow/blob/main/dev/breeze/doc/03_developer_tasks.rst
Compose

airflow/airflow-core/docs/howto/docker-compose/docker-compose.yaml

Line 155 in 84cd1d6

test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob --hostname "$${HOSTNAME}"']

Helm

airflow/chart/templates/dag-processor/dag-processor-deployment.yaml

Line 21 in 84cd1d6

## Airflow Dag Processor Deployment

[dheerajturaga]

@bugraoz93 , We dont use docker at all (and have no access to it either). We use virtualenvs. As mentioned, the setup here is pretty light weight where airflow cli comes from a local venv where airflow is installed. We just have these three env variables that go with it.

AIRFLOW_HOME = '<dags_repo>/run'
AIRFLOW__CORE__DAGS_FOLDER = <dags_repo>
AIRFLOW__CORE__LOAD_EXAMPLES = false

Being able to list dags and list import errors without having any services running is a valid and wide spread usecase. This is how folks sanity check their dags and maybe even run airflow standalone for a quick dry run before they deploy to the central airflow service

[bugraoz93]

@bugraoz93 , We dont use docker at all (and have no access to it either). We use virtualenvs. As mentioned, the setup here is pretty light weight where airflow cli comes from a local venv where airflow is installed. We just have these three env variables that go with it.

AIRFLOW_HOME = '<dags_repo>/run'
AIRFLOW__CORE__DAGS_FOLDER = <dags_repo>
AIRFLOW__CORE__LOAD_EXAMPLES = false

Being able to list dags and list import errors without having any services running is a valid and wide spread usecase. This is how folks sanity check their dags and maybe even run airflow standalone for a quick dry run before they deploy to the central airflow service

I am saying that Dag Processors should be running when deploying Airflow 3.0+ to any environment. I don’t know the details of your setup but when you say local, I understand as it is local machine you are using for development if that is running the airflow api-server, scheduler or triggerer in a venv, then yes Dag Processors should be running there too. Even in on-premise server or docker that shouldn't really make any difference in this scenario. If venv is remote and you are connecting, not in your personal local machines but somewhere in a server then it should be running there

[bugraoz93]

airflow/airflow-core/src/airflow/cli/commands/standalone_command.py

Line 79 in bdf4124

self.subcommands["dag-processor"] = SubCommand(

airflow standalone also should run Dag Processors too