-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exclude pip from the app image #255
Labels
Comments
edmorley
added a commit
that referenced
this issue
Aug 30, 2024
pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. This is possible now that the user site-packages is no longer being used for app dependencies, after the switch to venvs in #257. pip being in its own layer has the following advantages: 1. We can more easily exclude pip from the build/run images when using other packages managers (such as for the upcoming Poetry support). 2. A change in pip version no longer unnecessarily invalidates the Python layer. 3. In the future we could more easily exclude pip from the run image entirely, should we wish (see #255). This has been split out of the Poetry PR for easier review. Closes #254. GUS-W-16616956.
edmorley
added a commit
that referenced
this issue
Aug 30, 2024
pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. This is possible now that the user site-packages is no longer being used for app dependencies, after the switch to venvs in #257. pip being in its own layer has the following advantages: 1. We can more easily exclude pip from the build/run images when using other packages managers (such as for the upcoming Poetry support). 2. A change in pip version no longer unnecessarily invalidates the Python layer. 3. In the future we could more easily exclude pip from the run image entirely, should we wish (see #255). This has been split out of the Poetry PR for easier review. Closes #254. GUS-W-16616956.
edmorley
added a commit
that referenced
this issue
Aug 30, 2024
pip is now installed into its own layer (as a user site-packages install) instead of into system site-packages in the Python layer. This is possible now that the user site-packages location is no longer being used for app dependencies, after the switch to venvs in #257. pip being in its own layer has the following advantages: 1. We can more easily exclude pip from the build/run images when using other packages managers (such as for the upcoming Poetry support). 2. A change in pip version no longer unnecessarily invalidates the Python layer. 3. In the future we could more easily exclude pip from the run image entirely, should we wish (see #255). This has been split out of the Poetry PR for easier review. Closes #254. GUS-W-16616956.
edmorley
added a commit
that referenced
this issue
Sep 4, 2024
After #254, pip is now installed into its own layer rather than into the system site-packages directory inside the Python layer. This means its now possible to exclude pip from the final app image, by making the pip layer be a build-only layer. Excluding pip from the final app image: - Prevents several classes of user error/confusion/bad app design patterns seen in support tickets (see #255 for more details). - Reduces app image supply chain surface area. - Reduces app image size by 13 MB and layer count by 1, meaning less to have to push to the remote registry. - Matches the approach used for Poetry, where we don't make Poetry available at run-time either. Users that need pip at run-time for a temporary debugging task can run `python -m ensurepip --default-pip` in the container at run-time to make it available again (this command doesn't even have to download anything - it uses the pip bundled with Python). Or if pip is an actual run-time dependency of the app, then the app can add `pip` to its `requirements.txt` (which much more clearly conveys the requirements of the app, and also allows the app to pick what pip version it needs at run-time). Closes #255.
edmorley
added a commit
that referenced
this issue
Sep 4, 2024
After #254, pip is now installed into its own layer rather than into the system site-packages directory inside the Python layer. This means its now possible to exclude pip from the final app image, by making the pip layer be a build-only layer. Excluding pip from the final app image: - Prevents several classes of user error/confusion/bad app design patterns seen in support tickets (see #255 for more details). - Reduces app image supply chain surface area. - Reduces app image size by 13 MB and layer count by 1, meaning less to have to push to the remote registry. - Matches the approach used for Poetry, where we don't make Poetry available at run-time either. Users that need pip at run-time for a temporary debugging task can run `python -m ensurepip --default-pip` in the container at run-time to make it available again (this command doesn't even have to download anything - it uses the pip bundled with Python). Or if pip is an actual run-time dependency of the app, then the app can add `pip` to its `requirements.txt` (which much more clearly conveys the requirements of the app, and also allows the app to pick what pip version it needs at run-time). Should we find that pip's absence causes confusion in the future, we could always add a wrapper/shim `pip` script in the app image which does something like: ``` echo "pip isn't installed at run-time, if you need it temporarily run 'python -m ensurepip --default-pip' to install it" exit 1 ``` ...to improve discoverability. We'll also document pip (and Poetry) being available at build-time only in the docs that will be added by #11. Closes #255.
So I'm leaning towards trying to do this sooner rather than later, since it's a change that really needs to be made whilst CNBs on Heroku are still in preview, rather than after they GA (reach General Availability). Ref the open questions:
|
edmorley
added a commit
that referenced
this issue
Sep 9, 2024
After #254, pip is now installed into its own layer rather than into the system site-packages directory inside the Python layer. This means its now possible to exclude pip from the final app image, by making the pip layer be a build-only layer. Excluding pip from the final app image: - Prevents several classes of user error/confusion/bad app design patterns seen in support tickets (see #255 for more details). - Reduces app image supply chain surface area. - Reduces app image size by 13 MB and layer count by 1, meaning less to have to push to the remote registry. - Matches the approach used for Poetry, where we don't make Poetry available at run-time either. Users that need pip at run-time for a temporary debugging task can run `python -m ensurepip --default-pip` in the container at run-time to make it available again (this command doesn't even have to download anything - it uses the pip bundled with Python). Or if pip is an actual run-time dependency of the app, then the app can add `pip` to its `requirements.txt` (which much more clearly conveys the requirements of the app, and also allows the app to pick what pip version it needs at run-time). Should we find that pip's absence causes confusion in the future, we could always add a wrapper/shim `pip` script in the app image which does something like: ``` echo "pip isn't installed at run-time, if you need it temporarily run 'python -m ensurepip --default-pip' to install it" exit 1 ``` ...to improve discoverability. We'll also document pip (and Poetry) being available at build-time only in the docs that will be added by #11. Closes #255. GUS-W-16697386.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Once pip is installed into its own layer (see #254), it will be possible to make the layer be a build time only layer (and thus exclude pip from the run image) should we wish.
Doing so would:
pip install
, not realising that each dyno is separate and has an ephemeral filesystem (this happens more than one might expect)pip install
packages at app boot (eg viaweb: pip install foo && ...
in theirProcfile
) - which leads to slow app boot, non-determinism (each time a dyno boots a different version of a package or its transitive dependencies can be installed), and reliability issues (if PyPI is down the app can't start)Users that needed pip at run-time for a temporary task could use
python -m ensurepip --default-pip
to make it available again (this command doesn't even have to download anything - it uses the pip bundled with Python).Or if pip is an actual run-time dependency of the app, then the app can add
pip
to itsrequirements.txt
(which much more clearly conveys the requirements of the app, and also allows the app to pick what pip version it needs at run-time - something that isn't possible with the pip installed by the buildpack).Open questions:
pip run
type command, so I'm struggling to come up with any use-cases where it might be used as part of normal day to day app operation)pip
script that does something likeecho "pip isn't installed at run-time, if you need it temporarily run 'python -m ensurepip --default-pip' to install it" && exit 1
to improve discoverability?GUS-W-16697386.
The text was updated successfully, but these errors were encountered: