Add 'breeze k8s dev' command for hot-reloading DAGs and core sources (#40005)

[nailo2c]

Closes: #40005

Why

Based on my experience resolving k8s-related bugs, it's quite inconvenience to build and upload a k8s image when I only make a small change.

How

After some research, I decided integrate Skaffold into breeze k8s to improve the development experience.

I added a new command, breeze k8s dev, to sync Airflow source code to K8s pods in seconds.

What

1. /dags folder synchronization

airflow_40005_demo_output.mp4

2. Airflow Core synchronization

2.1. Scheduler

Exec into the pod with kubectl exec -n airflow -c scheduler -it airflow-scheduler-0 -- /bin/bash, and modify the log message to "Starting the scheduler [hot-reload]":

airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py

Line 1184 in ee5e21b

self.log.info("Starting the scheduler")

Use the following command to verify the synchronization: grep -n "Starting the scheduler" airflow-core/src/airflow/jobs/scheduler_job_runner.py

2.2. Triggerer

I used kubectl logs -n airflow -l component=triggerer -c triggerer -f to observe the logs. After I modified the file at airflow-core/src/airflow/jobs/triggerer_job_runner.py, the logs showed:

2025-12-23T02:04:41.298585Z [info     ] Detected changes: {(<Change.deleted: 3>, '/opt/airflow/airflow-core/src/airflow/cli/commands/dag_processor_command.py'), (<Change.added: 1>, '/opt/airflow/airflow-core/src/airflow/cli/commands/dag_processor_command.py')} [airflow.cli.hot_reload] loc=hot_reload.py:187
2025-12-23T02:04:41.304871Z [info     ] Reloading...                   [airflow.cli.hot_reload] loc=hot_reload.py:188

2.3. Dag-Processor

I used kubectl logs -n airflow -l component=dag-processor -c dag-processor -f to observe the logs. After I modified the file at airflow-core/src/airflow/cli/commands/dag_processor_command.py, the logs showed:

2025-12-23T02:04:40.964319Z [info     ] Detected changes: {(<Change.deleted: 3>, '/opt/airflow/airflow-core/src/airflow/cli/commands/dag_processor_command.py'), (<Change.added: 1>, '/opt/airflow/airflow-core/src/airflow/cli/commands/dag_processor_command.py')} [airflow.cli.hot_reload] loc=hot_reload.py:187
2025-12-23T02:04:40.970835Z [info     ] Reloading...                   [airflow.cli.hot_reload] loc=hot_reload.py:188

[potiuk]

WOWOWOWOWOW !

[potiuk]

One comment here - (might be follow-up) - if we run airflow commands in the chart with --dev flags , we could also simply synchronize airflow's source code - not only Dags. We already have built-in hot-reload in airflow components when they are run in --dev mode, so all we need is to sync the source change to the right folder.

[nailo2c]

Yes, exactly. That's why I chose Skaffold, it handles syncing the source code directly to the running containers without rebuilding images. Combined with the --dev flag, this should give us the fast iteration experience we need :)

[jason810496]

Yes, exactly. That's why I chose Skaffold, it handles syncing the source code directly to the running containers without rebuilding images. Combined with the --dev flag, this should give us the fast iteration experience we need :)

Yes, this is super nice feature for improving k8s test iteration! Very appreciate it!

The current PR should also close #44508.

[github-actions]

Backport failed to create: v3-1-test. View the failure log Run details

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker 9f874c3 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

[potiuk]

No backorting needed

[Lee-W]

#protm

[jason810496]

#protm

for me as well.
I forgot to add the tag while reviewing the PR.