Install app dependencies into a virtual environment #257
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
App dependencies are now installed into a virtual environment (aka venv or virtualenv) instead of into a custom user site-packages location. This: 1. Avoids user site-packages compatibility issues with some packages when using relocated Python (see #253) 2. Improves parity with how dependencies will be installed when using Poetry in the future (since Poetry doesn't support `--user`) 3. Unblocks being able to move pip into its own layer (see #254) This approach is possible since pip 22.3+ supports a new `--python` / `PIP_PYTHON` option which can be used to make pip operate against a different environment to the one in which it is installed. This allow us to continuing keeping pip in a separate layer to the app dependencies (currently the Python layer, but in a later PR pip will be moved to its own layer). Now that app dependencies are installed into a venv, we no longer need to make the system site-packages directory read-only to protect against later buildpacks installing into the wrong location. This has been split out of the Poetry PR for easier review. See also: - https://docs.python.org/3/library/venv.html - https://pip.pypa.io/en/stable/cli/pip/#cmdoption-python Closes #253. GUS-W-16616226.
edmorley
added a commit
that referenced
this pull request
Aug 30, 2024
pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. This is possible now that the user site-packages is no longer being used for app dependencies, after the switch to venvs in #257. pip being in its own layer has the following advantages: 1. We can more easily exclude pip from the build/run images when using other packages managers (such as for the upcoming Poetry support). 2. A change in pip version no longer unnecessarily invalidates the Python layer. 3. In the future we could more easily exclude pip from the run image entirely, should we wish (see #255). This has been split out of the Poetry PR for easier review. Closes #254. GUS-W-16616956.
joshwlewis
approved these changes
Aug 30, 2024
edmorley
added a commit
that referenced
this pull request
Aug 30, 2024
pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. This is possible now that the user site-packages is no longer being used for app dependencies, after the switch to venvs in #257. pip being in its own layer has the following advantages: 1. We can more easily exclude pip from the build/run images when using other packages managers (such as for the upcoming Poetry support). 2. A change in pip version no longer unnecessarily invalidates the Python layer. 3. In the future we could more easily exclude pip from the run image entirely, should we wish (see #255). This has been split out of the Poetry PR for easier review. Closes #254. GUS-W-16616956.
edmorley
added a commit
that referenced
this pull request
Aug 30, 2024
pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. This is possible now that the user site-packages location is no longer being used for app dependencies, after the switch to venvs in #257. pip being in its own layer has the following advantages: 1. We can more easily exclude pip from the build/run images when using other packages managers (such as for the upcoming Poetry support). 2. A change in pip version no longer unnecessarily invalidates the Python layer. 3. In the future we could more easily exclude pip from the run image entirely, should we wish (see #255). This has been split out of the Poetry PR for easier review. Closes #254. GUS-W-16616956.
This was referenced Aug 30, 2024
Merged
edmorley
added a commit
that referenced
this pull request
Sep 3, 2024
The Python package manager Poetry is now supported for installing app dependencies: https://python-poetry.org To use Poetry apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). Apps must only have one package manager file (either `requirements.txt` or `poetry.lock`, but not both) otherwise the buildpack will abort the build with an error (which will help prevent some of the types of support tickets we see in the classic buildpack). Poetry is installed into a build-only layer, so is not available at run-time to reduce image size. The app dependencies are installed into a virtual environment (the same as for pip, after #257), which is on `PATH` so does not need explicit activation when using the app image. As such, use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not explicitly installed, since Poetry includes its own bundled copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). Both the Poetry and app dependencies layers are cached, however, the Poetry download/wheel cache is not cached, since using it is slower than caching the dependencies layer (for more details see the comments on `poetry_dependencies::install_dependencies`). The `poetry install --sync` command is run using `--only main` so as to only install the main dependencies group and not any other groups (such as test/dev/... groups). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups Work that will be handled later: - Support for selecting Python version via `tool.poetry.dependencies.python`: #260 - Build output and error messages polish/CX review (this will be performed when switching the buildpack to the new logging style). - More detailed user-facing docs: #11 Closes #7. GUS-W-9607867. GUS-W-9608286. GUS-W-9608295.
edmorley
added a commit
that referenced
this pull request
Sep 4, 2024
The Python package manager Poetry is now supported for installing app dependencies: https://python-poetry.org To use Poetry, apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). Apps must only have one package manager file (either `requirements.txt` or `poetry.lock`, but not both) otherwise the buildpack will abort the build with an error (which will help prevent some of the types of support tickets we see in the classic buildpack with users unknowingly mixing and matching pip + Pipenv). Poetry is installed into a build-only layer (to reduce the final app image size), so is not available at run-time. The app dependencies are installed into a virtual environment (the same as for pip after #257, for the reasons described in #253), which is on `PATH` so does not need explicit activation when using the app image. As such, use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not installed (possible thanks to #258), since Poetry includes its own internal vendored copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). Both the Poetry and app dependencies layers are cached, however, the Poetry download/wheel cache is not cached, since using it is slower than caching the dependencies layer (for more details see the comments on `poetry_dependencies::install_dependencies`). The `poetry install --sync` command is run using `--only main` so as to only install the main `[tool.poetry.dependencies]` dependencies group from `pyproject.toml`, and not any of the app's other dependency groups (such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`). I've marked this `semver: major` since in the (probably unlikely) event there are any early-adopter projects using this CNB that have both a `requirements.txt` and `poetry.lock` then this change will cause them to error (until one of the files is deleted). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups Work that will be handled later: - Support for selecting Python version via `tool.poetry.dependencies.python`: #260 - Build output and error messages polish/CX review (this will be performed when switching the buildpack to the new logging style). - More detailed user-facing docs: #11 Closes #7. GUS-W-9607867. GUS-W-9608286. GUS-W-9608295.
edmorley
added a commit
that referenced
this pull request
Sep 4, 2024
The Python package manager Poetry is now supported for installing app dependencies: https://python-poetry.org To use Poetry, apps must have a `poetry.lock` lockfile, which can be created by running `poetry lock` locally, after adding Poetry config to `pyproject.toml` (which can be done either manually or by using `poetry init`). Apps must only have one package manager file (either `requirements.txt` or `poetry.lock`, but not both) otherwise the buildpack will abort the build with an error (which will help prevent some of the types of support tickets we see in the classic buildpack with users unknowingly mixing and matching pip + Pipenv). Poetry is installed into a build-only layer (to reduce the final app image size), so is not available at run-time. The app dependencies are installed into a virtual environment (the same as for pip after #257, for the reasons described in #253), which is on `PATH` so does not need explicit activation when using the app image. As such, use of `poetry run` or `poetry shell` is not required at run-time to use dependencies in the environment. When using Poetry, pip is not installed (possible thanks to #258), since Poetry includes its own internal vendored copy that it will use instead (for the small number of Poetry operations for which it still calls out to pip, such as package uninstalls). Both the Poetry and app dependencies layers are cached, however, the Poetry download/wheel cache is not cached, since using it is slower than caching the dependencies layer (for more details see the comments on `poetry_dependencies::install_dependencies`). The `poetry install --sync` command is run using `--only main` so as to only install the main `[tool.poetry.dependencies]` dependencies group from `pyproject.toml`, and not any of the app's other dependency groups (such as test/dev groups, eg `[tool.poetry.group.test.dependencies]`). I've marked this `semver: major` since in the (probably unlikely) event there are any early-adopter projects using this CNB that have both a `requirements.txt` and `poetry.lock` then this change will cause them to error (until one of the files is deleted). Relevant Poetry docs: - https://python-poetry.org/docs/cli/#install - https://python-poetry.org/docs/configuration/ - https://python-poetry.org/docs/managing-dependencies/#dependency-groups Work that will be handled later: - Support for selecting Python version via `tool.poetry.dependencies.python`: #260 - Build output and error messages polish/CX review (this will be performed when switching the buildpack to the new logging style). - More detailed user-facing docs: #11 Closes #7. GUS-W-9607867. GUS-W-9608286. GUS-W-9608295.
Closed
edmorley
added a commit
that referenced
this pull request
Dec 17, 2024
Since it was only set to work around a uWSGI bug, and that bug does not occur when using a venv (which we now use as of #257). Closes #265. GUS-W-17454520.
edmorley
added a commit
that referenced
this pull request
Dec 17, 2024
Since it was only set to work around a uWSGI bug, and that bug does not occur when using a venv (which we now use as of #257). Closes #265. GUS-W-17454520.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
App dependencies are now installed into a Python virtual environment (aka venv / virtualenv) instead of into a custom user site-packages location.
This:
--userinstalls)This approach is possible since pip 22.3+ supports a new
--python/PIP_PYTHONoption which can be used to make pip operate against a different environment to the one in which it is installed. This allows us to continuing keeping pip in a separate layer to the app dependencies (currently the Python layer, but in a later PR pip will be moved to its own layer).Now that app dependencies are installed into a venv, we no longer need to make the system site-packages directory read-only to protect against later buildpacks installing into the wrong location.
Note: For a venv to work, it depends upon the
<venv_layer>/bin/pythonsymlink being earlier inPATHthan the main Python installation. To achieve that with CNBs, the venv's layer name must be alphabetically after the Python layer name. In addition, lifecycle 0.20.1+ is required, since earlier versions didn't implement the spec correctly during the execution of later buildpacks (see buildpacks/lifecycle#1393).This has been split out of the Poetry PR for easier review.
See also:
Closes #253.
GUS-W-16616226.