Simplify installing airflow from GitHub repo in breeze

[potiuk]

Breeze had the possibility of installing airflow from a branch of any Github repo - by providing VCS url, but it's been broken since the split of distributions. This PR adds capability of using owner/repo:branch as --use-airflow-version and the installation will be done using this GitHub repo.

Note! Such installation will NOT (currently) compile the assets so you will not be able to run api_server easily after such installation We might want to improve that in the future.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[gopidesupavan]

Cool thanks :)

[github-actions]

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 40ccc55 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

[jason810496]

Thank, Jarek! Blazing speed, we can now easily set up the unreleased v3-0-test for manual provider testing.

[amoghrajesh]

Late to the part Jarek, but this is a really nice improvement!

[potiuk]

Late to the part Jarek, but this is a really nice improvement!

Cool. I am glad It helps :D

[potiuk]

BTW. @amoghrajesh @jason810496 -> I thought about next steps:

1. add asset compilation after installing airflow (that might be tricky as currently we have no node in the image - so it might requires some tricks. Likely it will require downlading the sources, unpacking them and installing airflow with hatch build -c -t custom after installing node.

2. adding compat test in CI for v3-0-test - this way at ANY point in time we would know that all "main" providers are tested with "3.0.X in progress" at the tip of our branch.

Might be pretty nice addition to our test harness.

[jason810496]

There frontend not built before error when specifing with --use-airflow-version apache/airflow:main( even with start-airflow --dev-mode ).

breeze start-airflow --dev-mode --python 3.10 --backend postgres --integration localstack --mount-sources providers-and-tests --use-airflow-version apache/airflow:main

The Airflow start successfully in Breeze, but the frontend is not built before. So it will result in 500 internal error when accessing the UI due to missing the dist of frontend.

So I still switch to main branch and build distributions then switch back to branch of current PR and run with --use-airflow-version wheel ) as workaround:
#53821 (comment)

I will check where should I add the breeze compile-ui-assets or how to compile the frontend before start-airflow when I have time tomorrow.

[potiuk]

I will check where should I add the breeze compile-ui-assets or how to compile the frontend before start-airflow when I have time tomorrow.

Yes- that is a missing thing. The problem with it is that we currently run asset compilation outside of breeze - but the installation with --use-airflow-version already happens fully in breeze. And we have no node/yarn/pnpm installed in CI image.... The problem is that when we install from GitHub URL - we do not have local sources (which would be a different version) so even if you could run breeze command inside breeze (technically such inception shoud be possible- because we also forward docker socket to inside breeze) - what you really need is you need to:

1. install node + yarn + pnpm
2. find the directory in site-packages where airflow UI javascript sources are installed
3. run appropriate npm commands

Not a rocket science but it will take a bit of time to run, and it shoudl be done somewhere here: https://github.com/apache/airflow/blob/main/scripts/in_container/install_airflow_and_providers.py#L767 -> this is where the command to install stuff when --use-airflow-version is actually executed - and "airflow-core" should be installed right after - you should probably naed to add a flag in the InstallationSpec (compile_assets) and set it to true somewhere here - https://github.com/apache/airflow/blob/main/scripts/in_container/install_airflow_and_providers.py#L325 - because this is where we determine what "pip install" to add based on the --use-airflow flag.

[amoghrajesh]

adding compat test in CI for v3-0-test - this way at ANY point in time we would know that all "main" providers are tested with "3.0.X in progress" at the tip of our branch.

This does make a lot of sense yes.

[potiuk]

Converted those two tasks into two issues:

• Compile assets when installing airflow from Github Branch in Breeze #54453
• Add compat tests against "v3-X-test" branch tip for providers. #54454