Support uv in addition to pip when extending the Docker image
#37785
Assignees
Labels
Comments
yehoshuadimarsky
added
kind:feature
|
Yes. We just switched to |
|
BTW. It's not going to be very easy (and might need extra feature in |
|
Opened an issue for that astral-sh/uv#2077 |
potiuk
added a commit
that referenced
this issue
Feb 29, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. This allow two things: * we can use `uv` to built the production image, which gives 40%-50% saving for image build time. * user do not have to use `pip install --user` to install new packages * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used * you can use uv to install packages when you extend the image as a user * this PR switches to use `uv` by default for all prod images, but it adds a check if the image still builds with `pip`. Fixes: #37785
|
Found a way around it. For Airflow 2.9, uv should be fully supported - #37796 regardless if uv flag will be supported or not by The way how to test it:
export AIRFLOW_VERSION=2.8.2
export DOCKER_BUILDKIT=1
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--tag "my-tag:0.0.1"This should be built by You can also make another test: export AIRFLOW_VERSION=2.8.2
export DOCKER_BUILDKIT=1
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_USE_UV="true" \
--tag "my-tag:0.0.1"Should use Can you please test and give feedback @yehoshuadimarsky ? |
|
to be honest i'm not fully comfortable building the docker image using the source "code" Dockerfile in your PR, it's a bit long and overwhelming (>1500 line file). I'm in no rush, more of an enthusiast wanting to try out cool new tools, this is not a hard blocker for me now. I'm ok waiting until you finalize the workflow and have a finished image built. |
To be honest, there are good instructoins to run the build. It's exactly one Dockerfile to copy and single instruction to build it :) - and without people like you testihg it before I can only have hopes that it will solve your problem, but never a certainty. testing and trying before merge is the best way to see. If your tests confirm that it works, we can merge it with confidence, if it does not, then I have a chance to fix it before, it's of course up to you, but if it does not fit your needs after we release it in 2.9 - you have now opportunity to prevent it. You can make use of it or not, but then it it does not, then you might have only self-complaint that you have not tested it. But of course - it's up to you to help to test it or not. |
|
(and if it does not work we will find out about month from now - when 2.9 will be released BTW). |
potiuk
added a commit
that referenced
this issue
Mar 1, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. This allow two things: * we can use `uv` to built the production image, which gives 40%-50% saving for image build time. * user do not have to use `pip install --user` to install new packages * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used * you can use uv to install packages when you extend the image as a user * this PR switches to use `uv` by default for all prod images, but it adds a check if the image still builds with `pip` * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: #37785 Co-authored-by: Wei Lee <weilee.rx@gmail.com>
|
Just tried building the image on Windows following these steps:
$env:AIRFLOW_VERSION="2.8.2"
$env:DOCKER_BUILDKIT=1
docker build . --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" --tag "my-tag:0.0.1"
Is this because I'm on Windows not Unix? Edit: So tried using GitHub Codespaces, got this: |
|
Edit #3: Followed instructions here https://docs.docker.com/engine/install/debian/ to setup Docker with BuildKit on GH Codespaces, now I can build the image. It ran for a while, but then failed with this output: |
|
1. |
|
All right - I did more testing and I think It should work nicer now :) |
|
Just tried again, this time on my M1 Macbook (2020 Pro), doesn't work: ~/Documents/GitHub/ git clone git@github.com:apache/airflow.git airflow-upstream
Cloning into 'airflow-upstream'...
remote: Enumerating objects: 400720, done.
remote: Counting objects: 100% (662/662), done.
remote: Compressing objects: 100% (410/410), done.
remote: Total 400720 (delta 282), reused 573 (delta 244), pack-reused 400058
Receiving objects: 100% (400720/400720), 258.46 MiB | 10.83 MiB/s, done.
Resolving deltas: 100% (311879/311879), done.
~/Documents/GitHub/ cd airflow-upstream
~/Documents/GitHub/airflow-upstream/ [main] git checkout use-uv-for-prod-image
branch 'use-uv-for-prod-image' set up to track 'origin/use-uv-for-prod-image'.
Switched to a new branch 'use-uv-for-prod-image'Here's the command and the error output: ~/Documents/GitHub/airflow-upstream/ [use-uv-for-prod-image] export AIRFLOW_VERSION=2.8.2
export DOCKER_BUILDKIT=1
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--tag "my-tag:0.0.1"
[+] Building 75.3s (59/72) docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 72.08kB 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.4 0.5s
=> docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s
=> => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s
=> => sha256:8d5998862fafcacec68298ab34c9c28973251a5082836f31de0188cd914fae33 2.36kB / 2.36kB 0.0s
=> => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB 0.0s
=> => sha256:f259fd86ee23d7b4a2944432e3a09f45fa288bb84bfbd222b19b2abc7ddffbb9 528B / 528B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 3.12kB 0.0s
=> [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.5s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.0s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [main 1/18] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 4.1s
=> => resolve docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 0.1s
=> => sha256:515463c9b88285509388c9d127a8a56cf9602cd56ce59356c283b978897d7104 1.37kB / 1.37kB 0.0s
=> => sha256:054df0836f5230590646d8b7c62c54009a80c5c95c0c79e5430c3ef0f9c870a7 6.98kB / 6.98kB 0.0s
=> => sha256:f546e941f15b76df3d982d56985432b05bc065e3923fb35be25a4d33d5c0f911 29.16MB / 29.16MB 1.7s
=> => sha256:24935aba99a712c5a6efc86118762b0f914e1577a2c072dbc1fac88d3cfbc37c 3.32MB / 3.32MB 0.6s
=> => sha256:b7885ba641278df6e419c1c888a8981d86aeb940cc1e17c99d07acb42ddd8d57 13.74MB / 13.74MB 1.5s
=> => sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631dd54cfccc8 1.86kB / 1.86kB 0.0s
=> => sha256:ab172a9f49e99303f4e3f13978812bad85a6ccb74545e95f0a3c60dcf550642c 244B / 244B 0.8s
=> => sha256:f5995b23a281d55dc1aada4458690373f1fe4cf0ac1d711a01a4baaf10bff0bb 3.13MB / 3.13MB 1.3s
=> => extracting sha256:f546e941f15b76df3d982d56985432b05bc065e3923fb35be25a4d33d5c0f911 1.5s
=> => extracting sha256:24935aba99a712c5a6efc86118762b0f914e1577a2c072dbc1fac88d3cfbc37c 0.1s
=> => extracting sha256:b7885ba641278df6e419c1c888a8981d86aeb940cc1e17c99d07acb42ddd8d57 0.4s
=> => extracting sha256:ab172a9f49e99303f4e3f13978812bad85a6ccb74545e95f0a3c60dcf550642c 0.0s
=> => extracting sha256:f5995b23a281d55dc1aada4458690373f1fe4cf0ac1d711a01a4baaf10bff0bb 0.2s
=> [internal] settings cache mount permissions 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.0s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.1s
=> [internal] load build context 0.0s
=> => transferring context: 72.08kB 0.0s
=> [internal] preparing inline document 0.1s
=> [internal] preparing inline document 0.0s
=> [scripts 1/17] COPY <<EOF /install_os_dependencies.sh 0.1s
=> [scripts 2/17] COPY <<EOF /install_mysql.sh 0.0s
=> [scripts 3/17] COPY <<EOF /install_mssql.sh 0.0s
=> [scripts 4/17] COPY <<EOF /install_postgres.sh 0.0s
=> [scripts 5/17] COPY <<EOF /install_packaging_tools.sh 0.0s
=> [scripts 6/17] COPY <<EOF /install_airflow_dependencies_from_branch_tip.sh 0.0s
=> [scripts 7/17] COPY <<EOF /common.sh 0.0s
=> [scripts 8/17] COPY <<EOF /pip 0.0s
=> [scripts 9/17] COPY <<EOF /install_from_docker_context_files.sh 0.0s
=> [scripts 10/17] COPY <<EOF /get_package_specs.py 0.0s
=> [scripts 11/17] COPY <<EOF /install_airflow.sh 0.0s
=> [scripts 12/17] COPY <<EOF /install_additional_dependencies.sh 0.0s
=> [scripts 13/17] COPY <<EOF /create_prod_venv.sh 0.0s
=> [scripts 14/17] COPY <<EOF /create_prod_venv.sh 0.0s
=> [scripts 15/17] COPY <<EOF /entrypoint_prod.sh 0.0s
=> [scripts 16/17] COPY <<EOF /clean-logs.sh 0.0s
=> [scripts 17/17] COPY <<EOF /airflow-scheduler-autorestart.sh 0.0s
=> [main 2/18] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.1s
=> [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 43.5s
=> [main 3/18] RUN bash /scripts/docker/install_os_dependencies.sh runtime 21.6s
=> [main 4/18] COPY --from=scripts common.sh /scripts/docker/ 0.0s
=> [main 5/18] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.0s
=> [main 6/18] RUN bash /scripts/docker/install_mysql.sh prod && bash /scripts/docker/install_mssql.sh prod && bash /scripts/docker 19.6s
=> [airflow-build-image 4/16] COPY --from=scripts common.sh /scripts/docker/ 0.0s
=> [airflow-build-image 5/16] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.0s
=> [airflow-build-image 6/16] RUN bash /scripts/docker/install_mysql.sh dev && bash /scripts/docker/install_mssql.sh dev && bash /s 15.2s
=> [airflow-build-image 7/16] RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password --quiet "airflow" - 0.3s
=> [airflow-build-image 8/16] COPY --chown=50000:0 Dockerfile /docker-context-files 0.0s
=> [airflow-build-image 9/16] RUN if [[ -f /docker-context-files/pip.conf ]]; then mkdir -p /home/airflow/.config/pip; cp /d 0.3s
=> [airflow-build-image 10/16] COPY --from=scripts common.sh install_packaging_tools.sh install_airflow_dependencies_from_branch_tip.sh 0.1s
=> [airflow-build-image 11/16] RUN bash /scripts/docker/install_packaging_tools.sh; bash /scripts/docker/create_prod_venv.sh; if [[ f 9.2s
=> [airflow-build-image 12/16] COPY --chown=airflow:0 Dockerfile /Dockerfile 0.0s
=> [airflow-build-image 13/16] WORKDIR /opt/airflow 0.0s
=> [airflow-build-image 14/16] COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh install_additional_dependenci 0.0s
=> ERROR [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-arm64-9,target=/tmp/.cache/pip,uid=50000 if 0.6s
------
> [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-arm64-9,target=/tmp/.cache/pip,uid=50000 if [[ false == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find "/home/airflow/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find "/home/airflow/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find "/home/airflow/.local" -executable ! -type l -print0 | xargs --null chmod g+x; find "/home/airflow/.local" ! -type l -print0 | xargs --null chmod g+rw:
0.097
0.097 Using 'pip' to install Airflow
0.097
0.117
0.117 Downloading constraints from https://raw.githubusercontent.com/apache/airflow/constraints-2.8.2/constraints-3.8.txt to /home/airflow/constraints.txt
0.117
0.242 PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/mssql-tools/bin
0.630 Installed pip: pip 24.0 from /home/airflow/.local/lib/python3.8/site-packages/pip (python 3.8): /home/airflow/.local/bin/pip
0.630 Using 'pip' to install Airflow
0.630
0.630 ERROR! You can only use --editable flag when installing airflow from sources!
0.630 Current installation method is 'apache-airflow and should be '.'
------
Dockerfile:1577
--------------------
1576 | # hadolint ignore=SC2086, SC2010, DL3042
1577 | >>> RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \
1578 | >>> if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
1579 | >>> bash /scripts/docker/install_from_docker_context_files.sh; \
1580 | >>> fi; \
1581 | >>> if ! airflow version 2>/dev/null >/dev/null; then \
1582 | >>> bash /scripts/docker/install_airflow.sh; \
1583 | >>> fi; \
1584 | >>> if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
1585 | >>> bash /scripts/docker/install_additional_dependencies.sh; \
1586 | >>> fi; \
1587 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; \
1588 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; \
1589 | >>> # make sure that all directories and files in .local are also group accessible
1590 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" -executable ! -type l -print0 | xargs --null chmod g+x; \
1591 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" ! -type l -print0 | xargs --null chmod g+rw
1592 |
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == \"true\" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n \"${ADDITIONAL_PYTHON_DEPS}\" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local\" -executable ! -type l -print0 | xargs --null chmod g+x; find \"${AIRFLOW_USER_HOME_DIR}/.local\" ! -type l -print0 | xargs --null chmod g+rw" did not complete successfully: exit code: 1 |
|
Tried it now also in my GH Codespaces env, appears to be the same error. Setup: ~/Documents/GitHub/airflow-upstream/ [use-uv-for-prod-image] gh codespace ssh
? Choose codespace: apache/airflow (use-uv-for-prod-image): fictional orbit
Linux 53324cd94090 6.2.0-1019-azure #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@53324cd94090:/workspaces/airflow# git status
On branch use-uv-for-prod-image
Your branch is up to date with 'origin/use-uv-for-prod-image'.
nothing to commit, working tree clean
root@53324cd94090:/workspaces/airflow# git pull
remote: Enumerating objects: 184, done.
remote: Counting objects: 100% (182/182), done.
remote: Compressing objects: 100% (126/126), done.
remote: Total 184 (delta 69), reused 153 (delta 51), pack-reused 2
Receiving objects: 100% (184/184), 340.68 KiB | 9.21 MiB/s, done.
Resolving deltas: 100% (69/69), completed with 15 local objects.
From https://github.com/apache/airflow
+ 8773cb8d8c...e54302f882 use-uv-for-prod-image -> origin/use-uv-for-prod-image (forced update)
32bf0817f2..b921d4cd6f constraints-main -> origin/constraints-main
a7691b787b..73a632a5a0 main -> origin/main
* [new branch] update-helm-check-instructions -> origin/update-helm-check-instructions
* [new tag] helm-chart/1.13.0rc1 -> helm-chart/1.13.0rc1
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
root@53324cd94090:/workspaces/airflow# git pull --rebase
Successfully rebased and updated refs/heads/use-uv-for-prod-image.
root@53324cd94090:/workspaces/airflow# git pull
Already up to date.
root@53324cd94090:/workspaces/airflow# git status
On branch use-uv-for-prod-image
Your branch is up to date with 'origin/use-uv-for-prod-image'.
nothing to commit, working tree clean
root@53324cd94090:/workspaces/airflow# git remote -v
origin https://github.com/apache/airflow (fetch)
origin https://github.com/apache/airflow (push)
root@53324cd94090:/workspaces/airflow# export AIRFLOW_VERSION=2.8.2
root@53324cd94090:/workspaces/airflow# export DOCKER_BUILDKIT=1Command and error: root@53324cd94090:/workspaces/airflow# docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.8-slim-bookworm" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--tag "my-tag:0.0.1"
[+] Building 70.2s (61/74) docker:default
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 72.05kB 0.0s
=> [internal] load .dockerignore 0.2s
=> => transferring context: 3.08kB 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.4 0.4s
=> [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s
=> CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s
=> [internal] load metadata for docker.io/library/python:3.8-slim-bookworm 0.3s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> CACHED [internal] preparing inline document 0.0s
=> CACHED [internal] preparing inline document 0.2s
=> [internal] preparing inline document 0.9s
=> [internal] preparing inline document 1.0s
=> [airflow-build-image 1/16] FROM docker.io/library/python:3.8-slim-bookworm@sha256:23252009f10b4af8a8c90409c54a866473a251b001b74902f04631d 0.0s
=> [internal] preparing inline document 0.0s
=> [internal] preparing inline document 0.5s
=> [internal] preparing inline document 1.1s
=> [internal] preparing inline document 0.0s
=> [internal] preparing inline document 0.0s
=> CACHED [internal] settings cache mount permissions 0.0s
=> [internal] preparing inline document 0.0s
=> CACHED [internal] preparing inline document 0.0s
=> CACHED [internal] preparing inline document 0.0s
=> CACHED [internal] preparing inline document 0.0s
=> [internal] preparing inline document 1.1s
=> [internal] load build context 0.1s
=> => transferring context: 72.05kB 0.0s
=> [internal] preparing inline document 1.0s
=> [internal] preparing inline document 0.3s
=> CACHED [scripts 1/17] COPY <<EOF /install_os_dependencies.sh 0.0s
=> CACHED [scripts 2/17] COPY <<EOF /install_mysql.sh 0.0s
=> CACHED [scripts 3/17] COPY <<EOF /install_mssql.sh 0.0s
=> CACHED [scripts 4/17] COPY <<EOF /install_postgres.sh 0.0s
=> [scripts 5/17] COPY <<EOF /install_packaging_tools.sh 0.8s
=> [scripts 6/17] COPY <<EOF /install_airflow_dependencies_from_branch_tip.sh 0.3s
=> [scripts 7/17] COPY <<EOF /common.sh 0.3s
=> [scripts 8/17] COPY <<EOF /pip 0.3s
=> [scripts 9/17] COPY <<EOF /install_from_docker_context_files.sh 0.3s
=> [scripts 10/17] COPY <<EOF /get_package_specs.py 0.3s
=> [scripts 11/17] COPY <<EOF /install_airflow.sh 0.3s
=> [scripts 12/17] COPY <<EOF /install_additional_dependencies.sh 0.3s
=> [scripts 13/17] COPY <<EOF /create_prod_venv.sh 0.3s
=> [scripts 14/17] COPY <<EOF /create_prod_venv.sh 0.3s
=> [scripts 15/17] COPY <<EOF /entrypoint_prod.sh 0.3s
=> [scripts 16/17] COPY <<EOF /clean-logs.sh 0.3s
=> [scripts 17/17] COPY <<EOF /airflow-scheduler-autorestart.sh 0.3s
=> CACHED [airflow-build-image 2/16] COPY --from=scripts install_os_dependencies.sh /scripts/docker/ 0.0s
=> CACHED [airflow-build-image 3/16] RUN bash /scripts/docker/install_os_dependencies.sh dev 0.0s
=> [airflow-build-image 4/16] COPY --from=scripts common.sh /scripts/docker/ 0.5s
=> CACHED [main 3/18] RUN bash /scripts/docker/install_os_dependencies.sh runtime 0.0s
=> [main 4/18] COPY --from=scripts common.sh /scripts/docker/ 0.6s
=> [airflow-build-image 5/16] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.5s
=> [main 5/18] COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/ 0.6s
=> [airflow-build-image 6/16] RUN bash /scripts/docker/install_mysql.sh dev && bash /scripts/docker/install_mssql.sh dev && bash /s 40.7s
=> [main 6/18] RUN bash /scripts/docker/install_mysql.sh prod && bash /scripts/docker/install_mssql.sh prod && bash /scripts/docker 40.6s
=> [airflow-build-image 7/16] RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password --quiet "airflow" - 1.3s
=> [airflow-build-image 8/16] COPY --chown=50000:0 Dockerfile /docker-context-files 0.3s
=> [airflow-build-image 9/16] RUN if [[ -f /docker-context-files/pip.conf ]]; then mkdir -p /home/airflow/.config/pip; cp /d 0.6s
=> [airflow-build-image 10/16] COPY --from=scripts common.sh install_packaging_tools.sh install_airflow_dependencies_from_branch_tip.sh 0.5s
=> [airflow-build-image 11/16] RUN bash /scripts/docker/install_packaging_tools.sh; bash /scripts/docker/create_prod_venv.sh; if [[ 15.5s
=> [airflow-build-image 12/16] COPY --chown=airflow:0 Dockerfile /Dockerfile 0.4s
=> [airflow-build-image 13/16] WORKDIR /opt/airflow 0.3s
=> [airflow-build-image 14/16] COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh install_additional_dependenci 0.5s
=> ERROR [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-amd64-9,target=/tmp/.cache/pip,uid=50000 if 1.6s
------
> [airflow-build-image 15/16] RUN --mount=type=cache,id=python:3.8-slim-bookworm-24.0-amd64-9,target=/tmp/.cache/pip,uid=50000 if [[ false == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find "/home/airflow/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find "/home/airflow/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find "/home/airflow/.local" -executable ! -type l -print0 | xargs --null chmod g+x; find "/home/airflow/.local" ! -type l -print0 | xargs --null chmod g+rw:
0.512
0.512 Using 'pip' to install Airflow
0.512
0.546
0.546 Downloading constraints from https://raw.githubusercontent.com/apache/airflow/constraints-2.8.2/constraints-3.8.txt to /home/airflow/constraints.txt
0.546
0.803 PATH=/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/mssql-tools/bin
1.428 Installed pip: pip 24.0 from /home/airflow/.local/lib/python3.8/site-packages/pip (python 3.8): /home/airflow/.local/bin/pip
1.428 Using 'pip' to install Airflow
1.428
1.428 ERROR! You can only use --editable flag when installing airflow from sources!
1.428 Current installation method is 'apache-airflow and should be '.'
------
Dockerfile:1577
--------------------
1576 | # hadolint ignore=SC2086, SC2010, DL3042
1577 | >>> RUN --mount=type=cache,id=$PYTHON_BASE_IMAGE-$AIRFLOW_PIP_VERSION-$TARGETARCH-$PIP_CACHE_EPOCH,target=/tmp/.cache/pip,uid=${AIRFLOW_UID} \
1578 | >>> if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
1579 | >>> bash /scripts/docker/install_from_docker_context_files.sh; \
1580 | >>> fi; \
1581 | >>> if ! airflow version 2>/dev/null >/dev/null; then \
1582 | >>> bash /scripts/docker/install_airflow.sh; \
1583 | >>> fi; \
1584 | >>> if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
1585 | >>> bash /scripts/docker/install_additional_dependencies.sh; \
1586 | >>> fi; \
1587 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -name '*.pyc' -print0 | xargs -0 rm -f || true ; \
1588 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local/" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; \
1589 | >>> # make sure that all directories and files in .local are also group accessible
1590 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" -executable ! -type l -print0 | xargs --null chmod g+x; \
1591 | >>> find "${AIRFLOW_USER_HOME_DIR}/.local" ! -type l -print0 | xargs --null chmod g+rw
1592 |
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == \"true\" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; fi; if ! airflow version 2>/dev/null >/dev/null; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n \"${ADDITIONAL_PYTHON_DEPS}\" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -name '*.pyc' -print0 | xargs -0 rm -f || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local/\" -type d -name '__pycache__' -print0 | xargs -0 rm -rf || true ; find \"${AIRFLOW_USER_HOME_DIR}/.local\" -executable ! -type l -print0 | xargs --null chmod g+x; find \"${AIRFLOW_USER_HOME_DIR}/.local\" ! -type l -print0 | xargs --null chmod g+rw" did not complete successfully: exit code: 1 |
|
Should be fine now. Missed a case there. |
|
That works! Thanks |
|
Thanks for helping with tests. Also |
potiuk
added a commit
to potiuk/airflow
that referenced
this issue
Mar 3, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: apache#37785 Fixes: apache#37815
potiuk
added a commit
that referenced
this issue
Mar 3, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: #37785 Fixes: #37815
|
Not sure if this matters this much, but building the Docker image on Windows still fails, even after I ran |
potiuk
added a commit
that referenced
this issue
Mar 5, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: #37785 Fixes: #37815 Update contributing-docs/testing/k8s_tests.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update contributing-docs/testing/k8s_tests.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update scripts/docker/install_airflow.sh Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/changelog.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com>
potiuk
added a commit
that referenced
this issue
Mar 6, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: #37785 Fixes: #37815 Update contributing-docs/testing/k8s_tests.rst Update contributing-docs/testing/k8s_tests.rst Update docs/docker-stack/build.rst Update docs/docker-stack/build.rst Update docs/docker-stack/build.rst Update docs/docker-stack/build.rst Update docs/docker-stack/build.rst Update scripts/docker/install_airflow.sh Update docs/docker-stack/changelog.rst Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <onikolas@amazon.com>
|
You likely need to check it out from scratch for autocrlf to be effective. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Description
I am trying to use the new phenomenal tool
uvto add python packages on top of the base Airflow image. I kept running into the issue ofuvnot supporting non-virtual-envs, but this is now supported with version 0.1.12Here is my sample Dockerfile. I cannot get it to work with either of these new flags:
--pythonor--system.In both cases, I get this error:
I thought, ok, let me run it as
USER root, but then I get an error about installingpippackages as root, from here: https://airflow.apache.org/docs/docker-stack/build.html#adding-new-pypi-packages-individuallyAny suggestions on using
uvwith the Airflow Docker image?Thanks
Use case/motivation
No response
Related issues
I see that Airflow has already adopted
uvinternally for its CI here and see (https://github.com/apache/airflow/issues?q=uv), so I think this should be doable for the external Docker image as wellAre you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: