Skip to content

Commit

Permalink
Make Breeze installation and reinstallation support both uv and pipx
Browse files Browse the repository at this point in the history
So far `breeze` fully supported only `pipx` installation. For
example it would reinstall itself automatically with pipx if you
attempted to run it from another workspace/checked out repository
of Airflow, and it only provided instructions for pipx.

With this PR:

* the `uv tool` is preferred way to install breeze
* the `pipx` is PSF-governance managed alternative
* breeze will reinstall itself using uv if it has been installed with uv
  before when it is run from a different workspace or different
  airflow repo checked out in another folder

Also documentation is updated to make `uv` the recommended tool and
describing how to install it - with `pipx` provided as an alternative.
  • Loading branch information
potiuk committed Nov 2, 2024
1 parent 59ea748 commit edd26a4
Show file tree
Hide file tree
Showing 7 changed files with 66 additions and 33 deletions.
25 changes: 20 additions & 5 deletions dev/breeze/doc/01_installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -151,13 +151,28 @@ Docker in WSL 2
If VS Code is installed on the Windows host system then in the WSL Linux Distro
you can run ``code .`` in the root directory of you Airflow repo to launch VS Code.

The pipx tool
--------------
The uv tool
-----------

We are recommending to use the ``uv`` tool to manage your virtual environments and generally as a swiss-knife
of your Python environment (it supports installing various versions of Python, creating virtual environments,
installing packages, managing workspaces and running development tools.).

Installing ``uv`` is described in the `uv documentation <https://docs.astral.sh/uv/getting-started/installation/>`_.
We highly recommend using ``uv`` to manage your Python environments, as it is very comprehensive,
easy to use, it is faster than any of the other tools availables (wey faster!) and has a lot of features
that make it easier to work with Python.

Alternative for breeze installation: the pipx tool
--------------------------------------------------

We are using ``pipx`` tool to install and manage Breeze. The ``pipx`` tool is created by the creators
However - we do not want to be entirely dependent on ``uv`` as it is a software governed by e VC-backed vendor,
so we always want to provide open-source governed alternatives for our tools. If you can't or do not want to
use ``uv` we got you covered. Another too you can use to manage development tools (and ``breeze`` development
environment is Python-Software-Foundation managed ``pipx``. The ``pipx`` tool is created by the creators
of ``pip`` from `Python Packaging Authority <https://www.pypa.io/en/latest/>`_

Note that ``pipx`` >= 1.4.1 is used.
Note that ``pipx`` >= 1.4.1 should be used.

Install pipx

Expand All @@ -172,7 +187,7 @@ environments. This can be done automatically by the following command (follow in
pipx ensurepath
In Mac
In case ``pipx`` is not in your PATH, you can run it with Python module:

.. code-block:: bash
Expand Down
29 changes: 18 additions & 11 deletions dev/breeze/src/airflow_breeze/global_constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,15 @@
except ImportError:
get_console().print(
"\n[error]Breeze doesn't support Python version <=3.8\n\n"
"[warning]Use Python 3.9 and force reinstall breeze with pipx\n\n"
" pipx install --force -e ./dev/breeze\n"
"[warning]Use Python 3.9 and force reinstall breeze:"
""
" either with uv: \n\n"
" uv tool install --force --reinstall --editable ./dev/breeze\n\n"
""
" or with pipx\n\n"
" pipx install --force -e ./dev/breeze --python 3.9\n"
"\nTo find out more, visit [info]https://github.com/apache/airflow/"
"blob/main/dev/breeze/doc/01_installation.rst#the-pipx-tool[/]\n"
"blob/main/dev/breeze/doc/01_installation.rst[/]\n"
)
sys.exit(1)
from pathlib import Path
Expand Down Expand Up @@ -253,14 +258,16 @@ def all_helm_test_packages() -> list[str]:

@cache
def all_task_sdk_test_packages() -> list[str]:
return sorted(
[
candidate.name
for candidate in (AIRFLOW_SOURCES_ROOT / "task_sdk" / "tests").iterdir()
if candidate.is_dir() and candidate.name != "__pycache__"
]
)

try:
return sorted(
[
candidate.name
for candidate in (AIRFLOW_SOURCES_ROOT / "task_sdk" / "tests").iterdir()
if candidate.is_dir() and candidate.name != "__pycache__"
]
)
except FileNotFoundError:
return []

ALLOWED_TASK_SDK_TEST_PACKAGES = [
"all",
Expand Down
5 changes: 1 addition & 4 deletions dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -391,10 +391,7 @@ def create_virtualenv(force_venv_setup: bool) -> RunCommandResult:
"[info]You can uninstall breeze and install it again with earlier Python "
"version. For example:[/]\n"
)
get_console().print("pipx reinstall --python PYTHON_PATH apache-airflow-breeze\n")
get_console().print(
f"[info]PYTHON_PATH - path to your Python binary(< {higher_python_version_tuple})[/]\n"
)

get_console().print("[info]Then recreate your k8s virtualenv with:[/]\n")
get_console().print("breeze k8s setup-env --force-venv-setup\n")
sys.exit(1)
Expand Down
13 changes: 7 additions & 6 deletions dev/breeze/src/airflow_breeze/utils/path_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,9 @@ def reinstall_if_setup_changed() -> bool:
return False
if "apache-airflow-breeze" in e.msg:
print(
"""Missing Package `apache-airflow-breeze`.
Use `pipx install -e ./dev/breeze` to install the package."""
"""Missing Package `apache-airflow-breeze`. Please install it.\n
Use `uv tool install -e ./dev/breeze or `pipx install -e ./dev/breeze`
to install the package."""
)
return False
sources_hash = get_installation_sources_config_metadata_hash()
Expand Down Expand Up @@ -224,10 +225,10 @@ def get_used_airflow_sources() -> Path:
@cache
def find_airflow_sources_root_to_operate_on() -> Path:
"""
Find the root of airflow sources we operate on. Handle the case when Breeze is installed via `pipx` from
a different source tree, so it searches upwards of the current directory to find the right root of
airflow directory we are actually in. This **might** be different than the sources of Airflow Breeze
was installed from.
Find the root of airflow sources we operate on. Handle the case when Breeze is installed via
`pipx` or `uv tool` from a different source tree, so it searches upwards of the current directory
to find the right root of airflow directory we are actually in. This **might** be different
than the sources of Airflow Breeze was installed from.
If not found, we operate on Airflow sources that we were installed it. This handles the case when
we run Breeze from a "random" directory.
Expand Down
7 changes: 4 additions & 3 deletions dev/breeze/src/airflow_breeze/utils/python_versions.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ def check_python_version(release_provider_packages: bool = False):
get_console().print(
"[warning]Please reinstall Breeze using Python 3.9 - 3.11 environment because not all "
"provider packages support Python 3.12 yet.[/]\n\n"
"For example:\n\n"
"pipx uninstall apache-airflow-breeze\n"
"pipx install --python $(which python3.9) -e ./dev/breeze --force\n"
"If you are using uv:\n\n"
" uv tool install --force --reinstall --python 3.9 -e ./dev/breeze\n\n"
"If you are using pipx:\n\n"
" pipx install --python $(which python3.9) --force -e ./dev/breeze\n"
)
sys.exit(1)
13 changes: 11 additions & 2 deletions dev/breeze/src/airflow_breeze/utils/reinstall.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,24 @@

def reinstall_breeze(breeze_sources: Path, re_run: bool = True):
"""
Reinstalls Breeze from specified sources.
Re-installs Breeze from specified sources.
:param breeze_sources: Sources where to install Breeze from.
:param re_run: whether to re-run the original command that breeze was run with.
"""
# First check if `breeze` is installed with uv and if it is, reinstall it using uv
# If not - we assume pipx is used and we reinstall it using pipx
# Note that we cannot use `pipx upgrade` here because we sometimes install
# Breeze from different sources than originally installed (i.e. when we reinstall airflow
# From the current directory.
get_console().print(f"\n[info]Reinstalling Breeze from {breeze_sources}\n")
subprocess.check_call(["pipx", "install", "-e", str(breeze_sources), "--force"])
result = subprocess.run(["uv", "tool", "list"], text=True, capture_output=True, check=False)
if result.returncode == 0:
if "apache-airflow-breeze" in result.stdout:
subprocess.check_call(
["uv", "tool", "install", "--force", "--reinstall", "-e", breeze_sources.as_posix()]
)
else:
subprocess.check_call(["pipx", "install", "-e", breeze_sources.as_posix(), "--force"])
if re_run:
# Make sure we don't loop forever if the metadata hash hasn't been updated yet (else it is tricky to
# run pre-commit checks via breeze!)
Expand Down
7 changes: 5 additions & 2 deletions scripts/ci/pre_commit/common_precommit_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,11 @@ def initialize_breeze_precommit(name: str, file: str):
if shutil.which("breeze") is None:
console.print(
"[red]The `breeze` command is not on path.[/]\n\n"
"[yellow]Please install breeze with `pipx install -e ./dev/breeze` from Airflow sources "
"and make sure you run `pipx ensurepath`[/]\n\n"
"[yellow]Please install breeze.\n"
"You can use uv with `uv tool install -e ./dev/breeze or "
"`pipx install -e ./dev/breeze`.\n"
"It will install breeze from Airflow sources "
"(make sure you run `pipx ensurepath` if you use pipx)[/]\n\n"
"[bright_blue]You can also set SKIP_BREEZE_PRE_COMMITS env variable to non-empty "
"value to skip all breeze tests."
)
Expand Down

0 comments on commit edd26a4

Please sign in to comment.