The setuptools and wheel versions are not fully pinned #1000

edmorley · 2020-07-22T10:42:28Z

As part of Python setup, the buildpack has to manage the installation of pip, setuptools and wheel, so that they are available for the later pip install of user-provided dependencies.

Changes in these packages can have a significant impact on the build (particularly since the buildpack has to manually modify site-packages as part of BUILD_DIR -> /app path rewriting, so is sadly sensitive to implementation details), so it's important that we pin to tested versions so that builds don't suddenly break after a new upstream release.

Equally as important as preventing new releases from being installed unintentionally, is ensuring that:

these packages in existing apps are upgraded if the buildpack pinned versions change (so they benefit from fixes, and to reduce the version matrix when reasoning about buildpack changes)
any user-provided custom versions of these packages are cleaned up if they are later removed from requirements.txt (to prevent their presence in the build cache causing hard to debug situations)

For pip, the buildpack currently does the right thing:

if pip is not present, a pinned version is installed (currently 9.0.2 for pipenv users, 19.1.1 for Python 3.4, and 20.0.2 otherwise)
if pip is present, the version is checked against the expected pinned version and upgraded/downgraded if necessary

However for setuptools, the behaviour is inconsistent:

for clean installs (new app, manually cleared build cache, Python/stack version upgrade), a pinned setuptools version is installed (currently 39.0.1)
for non-clean installs:
- if the pip version was changed since the last build, then setuptools is upgraded/downgraded to the pinned version (if necessary)
- otherwise, the setuptools version (or even the presence of setupools at all) is not checked/managed

And for wheel, the situation is worse:

for clean installs, the wheel version is not pinned so the latest is installed
for non-clean installs, so long as any version of wheel is installed, it's never upgraded/changed

The above leads to both unpredictability of installed versions, as well as confusing UX when performing other actions (eg failures after upgrading to a new Python patch version, that had nothing to do with the Python upgrade itself, but just happened to trigger installing newer setuptools/wheel etc).

We should pin all three to specific versions, and ensure that these versions are checked/managed during each build, and not just after a cache-clearing event.

See:

heroku-buildpack-python/bin/steps/python

Lines 150 to 161 in 156b07c

    
           # If a new Python has been installed or Pip isn't up to date: 
        
           if [ "$FRESH_PYTHON" ] || [[ ! $(pip --version) == *$PIP_UPDATE* ]]; then 
        
             puts-step "Installing pip" 
        
             # Remove old installations. 
        
             rm -fr /app/.heroku/python/lib/python*/site-packages/pip-* 
        
             rm -fr /app/.heroku/python/lib/python*/site-packages/setuptools-* 
        
             /app/.heroku/python/bin/python "$GETPIP_PY" pip=="$PIP_UPDATE" &> /dev/null 
        
             /app/.heroku/python/bin/pip install "$ROOT_DIR/vendor/setuptools-39.0.1-py2.py3-none-any.whl" &> /dev/null 
        
           fi

The text was updated successfully, but these errors were encountered:

The versions installed by the buildpack have been updated as follows: * pip: - If using Python 3.4: No change (already using the last to support 3.4) - If using pipenv: No change (need to update to a newer pipenv first) - For everything else: `20.0.2` -> `20.1.1` * setuptools: - If using Python 3.4: `39.0.1` -> `43.0.0` (latest for 3.4) - If using Python 2.7: `39.0.1` -> `44.1.1` (latest for 2.7) - For everything else: `39.0.1` -> `47.1.1` (until #1006 fixed) * wheel: - If using Python 3.4: `unpinned` -> `0.33.6` - For everything else: `unpinned` -> `0.34.2` This fixes #949 and fixes #1005, and means packages that rely on newer setuptools will now install successfully. Changelogs: https://pip.pypa.io/en/stable/news/ https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1 https://wheel.readthedocs.io/en/latest/news.html In addition: * Installed versions are now deterministic (fixes #1000, fixes #1003) * The build output now includes the versions used, making it easier to debug future upgrades (closes #939) * Errors during pip/setuptools/wheel install now correctly fail the build, and stderr is no longer sent to `/dev/null` (fixes #1002) * Setuptools is no longer installed twice (fixes #1001) * Everything that is downloaded is now used (fixes #999) * `--no-cache` and `--disable-version-check` are now used, saving unnecessary work and preventing creation of unwanted files in `/app` * The `PIP_UPDATE` env var no longer leaks into subprocesses. As part of fixing version pinning, we now use pip itself to determine whether the installed packages are up to date, since parsing pip's output is fragile (eg #1003). This means `pip install` is now called every time, however this is a no-op for repeat builds where the versions have not changed, since unless `--upgrade` is specified pip does not hit the index (PyPI) if requirements are satisfied. For the installation itself `get-pip.py` is no longer used, since: - It uses `--force-reinstall`, which is unnecessary here and would slow down repeat builds (given we call pip install every time now). Trying to work around this by using `get-pip.py` only for the initial install, and real pip for subsequent updates would mean we lose protection against cached broken installs, plus significantly increase the version combinations test matrix. - It means downloading pip twice (once embedded in `get-pip.py`, and again during the install, since `get-pip.py` can't install the embedded version directly) - We would still have to manage several versions of get-pip.py, to support older Pythons. We don't use `ensurepip` since: - Not all of the previously generated Python runtimes on S3 include it - We would still have to upgrade pip afterwards - The versions of pip/setuptools bundled with ensurepip differ greatly depending on Python version, and we could easily start using a CLI flag for the first pip install before upgrade that isn't supported on all versions, without even knowing it (unless we test against hundreds of Python archives). The new pip wheel assets on S3 were generated using: ``` $ pip download --no-cache pip==19.1.1 Collecting pip==19.1.1 Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB) Saved ./pip-19.1.1-py2.py3-none-any.whl Successfully downloaded pip $ pip download --no-cache pip==20.1.1 Collecting pip==20.1.1 Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB) Saved ./pip-20.1.1-py2.py3-none-any.whl Successfully downloaded pip $ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun (dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl (dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl $ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl ```

Before: - if `wheel` was not already installed, then `get-pip.py` would automatically install the latest version on PyPI, which is `0.34.2` (or `0.33.6` for Python 3.4). - if `wheel` was already installed, then it was left unchanged regardless of the version installed. Now: - if `wheel` is not already installed, then the same versions will be installed as before, except these versions are pinned and will now not change unexpectedly after future `wheel` releases. - if `wheel` is already installed, then it's upgraded/downgraded to the target version as needed. Partly addresses #1000, though this change only helps builds where the pip/setuptools/wheel install flow is triggered (currently only new apps or ones where Python was purged or pip was not the correct version). Since the wheel version is now known, it's output to the build log to ease debugging and for parity with pip/setuptools. The rest of #1000 will be fixed in later commits.

Previously the pip/setuptools/wheel install step was skipped so long as Python hadn't just been clean installed (ie so long as not a new app, emptied cache, Python upgrade, stack change) and pip was the expected version. This meant that setuptool/wheel could be the wrong version (or even just not installed at all), and this would not be corrected. Now, we now use pip itself to determine whether the installed packages are up to date, since parsing pip's output is fragile (eg #1003) and would be tedious given there would be three packages to check. Unfortunately `get-pip.py` uses `--force-reinstall` which means performing this step every time is not the no-op it would otherwise be, but this will be resolved by switching away from `get-pip.py` in the next commit. Fixes #1000. Fixes #1003. Closes #999.

Before: - if `wheel` was not already installed, then `get-pip.py` would automatically install the latest version on PyPI, which is `0.34.2` (or `0.33.6` for Python 3.4). - if `wheel` was already installed, then it was left unchanged regardless of the version installed. Now: - if `wheel` is not already installed, then the same versions will be installed as before, except these versions are pinned and will now not change unexpectedly after future `wheel` releases. - if `wheel` is already installed, then it's upgraded/downgraded to the target version as needed. Partly addresses #1000, though this change only helps builds where the pip/setuptools/wheel install flow is triggered (currently only new apps or ones where Python was purged or pip was not the correct version). Since the wheel version is now known, it's output to the build log to ease debugging and for parity with pip/setuptools. The rest of #1000 will be fixed in later commits.

Previously the pip/setuptools/wheel install step was skipped so long as Python hadn't just been clean installed (ie so long as not a new app, emptied cache, Python upgrade, stack change) and pip was the expected version. This meant that setuptool/wheel could be the wrong version (or even just not installed at all), and this would not be corrected. Now, we now use pip itself to determine whether the installed packages are up to date, since parsing pip's output is fragile (eg #1003) and would be tedious given there would be three packages to check. Unfortunately `get-pip.py` uses `--force-reinstall` which means performing this step every time is not the no-op it would otherwise be, but this will be resolved by switching away from `get-pip.py` in the next commit. Fixes #1000. Fixes #1003. Closes #999.

Before: - if `wheel` was not already installed, then `get-pip.py` would automatically install the latest version on PyPI, which is `0.34.2` (or `0.33.6` for Python 3.4). - if `wheel` was already installed, then it was left unchanged regardless of the version installed. Now: - if `wheel` is not already installed, then the same versions will be installed as before, except these versions are pinned and will now not change unexpectedly after future `wheel` releases. - if `wheel` is already installed, then it's upgraded/downgraded to the target version as needed. Partly addresses heroku#1000, though this change only helps builds where the pip/setuptools/wheel install flow is triggered (currently only new apps or ones where Python was purged or pip was not the correct version). Since the wheel version is now known, it's output to the build log to ease debugging and for parity with pip/setuptools. The rest of heroku#1000 will be fixed in later commits.

…u#1007) Previously the pip/setuptools/wheel install step was skipped so long as Python hadn't just been clean installed (ie so long as not a new app, emptied cache, Python upgrade, stack change) and pip was the expected version. This meant that setuptool/wheel could be the wrong version (or even just not installed at all), and this would not be corrected. Now, we now use pip itself to determine whether the installed packages are up to date, since parsing pip's output is fragile (eg heroku#1003) and would be tedious given there would be three packages to check. Unfortunately `get-pip.py` uses `--force-reinstall` which means performing this step every time is not the no-op it would otherwise be, but this will be resolved by switching away from `get-pip.py` in the next commit. Fixes heroku#1000. Fixes heroku#1003. Closes heroku#999.

edmorley self-assigned this Jul 22, 2020

edmorley added deterministic-builds bug labels Jul 22, 2020

edmorley mentioned this issue Jul 23, 2020

Improve pip/setuptools/wheel installation #1007

Merged

edmorley closed this as completed in #1007 Jul 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The setuptools and wheel versions are not fully pinned #1000

The setuptools and wheel versions are not fully pinned #1000

edmorley commented Jul 22, 2020 •

edited

Loading

The setuptools and wheel versions are not fully pinned #1000

The setuptools and wheel versions are not fully pinned #1000

Comments

edmorley commented Jul 22, 2020 • edited Loading

edmorley commented Jul 22, 2020 •

edited

Loading