Improve K8S tests iteration experience

[potiuk]

As also experienced by @ashb in #44463, our K8S tests are rather slow to iterate on and fix locally.

The way how to run those tests locally is explained in https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst#typical-testing-pattern-for-kubernetes-tests in detail - but there are a fewproblems to solve with them:

• The k8s image needs to use PROD base image to be build first and - especially when dependencies change - sometimes, but not always the PROD image needs to be rebuilt with --rebuild-base-image flag of breeze k8s build-k8s-image. This might cause the Helm chart to not work at all (for example it failed when asyncpg was not installed in the base image

• Currently when you modify the ./kubernetest_tests - i.e. tests that you run in the local k8s-dedicated virtulenv environment specially prepared for those tests, they can be re-run right after modification - which is great to iterate on the tests. However, this is NOT the case for "airflow" code that you modify and iterate on. While thisis understandable that if you modify helm chart, you need to redeploy the chart, you also need to rebuild the "k8s image" and re-build and re-upload the image to the kind cluster in order to iterate on your changes. This is SLOW LIKE HELL. Rebuilding the image, re-uploading it and re-deploying chart to reload airflow takes minutes.

So in order to fix it we can do several things:

• better detection when we need to re-build the image because of dependency change

• using uv to install dependencies in PROD image as well (currently it is only used by default in CI image) - this way we could likely even rebuild the image more often as it wil be faster

• add extra layer to add missing dependencies (with UV) in the K8S image:

  airflow/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py

  Line 602 in 0d6d5f2

  FROM {params.airflow_image_name_with_tag}

  - which means that we could continue to only rebuilding the k8s image (without --rebuild-base-image flag even if dependencies change)

• we could integrate solutions (or build our own) like Telepresence: https://www.telepresence.io/ where local sources of airflow would be synced (via ssh) with the sources in the k8s cluster (for example via local folder mounting or SSH) - this is a bit difficult to implement it to work in all circumstances (Linux/ MacOS/ WSL2) and all kind of docker engines our users might have (Docker Desktop, Orbstack, Podman etc.) - so doing it righ is a bit of a challenge. But that should be an ultimate way of iteration where changes to airflow and kuberntes providers could be followed by just pod restart and no need to build an upload new image.

[jason810496]

Taking inspiration from #47544.

How about using kubectl to copy the source code from the local machine to a PersistentVolume (airflow-source-code-volume), binding it to $AIRFLOW_HOME to include the Airflow core source code when running the breeze k8s reload command, and then rolling out a restart for all Airflow deployments?

Does this approach seem reasonable to you? I’d love to hear your thoughts!

[insomnes]

The PVC may also simplify flow for starting airflow in docker and only configure the executor to run pods in kind-cluster. To speed up the feedback loop even more.

I am not very proficient with breeze, but it would require changing the installation method to the editable one during k8s image building for this case. Because the current k8s flow uses run_build_production_image and it results in apache-airflow install method:

https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/params/build_prod_params.py#L143

So it would also require some kind of flag for the building step to add the ability to override it via ..

[potiuk]

Yeah. That sounds like a good direction. breeze k8s reload would do a rolling restart of everything after uploading the changed code. That sounds like a very nice approach.

And yeah it woudl likely have to have some modifications - but we could likely do it by having some "dev" mode in helm chart to be able to map the storage to the installed airflow sources.

[jason810496]

And yeah it woudl likely have to have some modifications - but we could likely do it by having some "dev" mode in helm chart to be able to map the storage to the installed airflow sources.

Sure, having "dev" mode to mount airflow source on installing the helm chart to k8s cluster with breeze k8s reload would be much cleaner ! ( Avoid having too much patching k8s components logic in Breeze )

I can work on this after some auth manager and API integration finish.

[jason810496]

Close this one as #59747 merged