Update pip to 20.3.4/21.3.1 (new dependency resolver) (#1259)

Update pip from 20.2.4 to: - 20.3.4 for Python 2.7 and 3.5 - 21.3.1 for Python 3.6+ Of note Pip 20.3+ includes the new dependency resolver (only enabled by default when using Python 3+). This new dependency resolver is more strict, see: https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020 https://pip.pypa.io/en/stable/topics/dependency-resolution/ Release notes: https://pip.pypa.io/en/stable/news/#v21-3-1 Changelog: pypa/pip@20.2.4...21.3.1 The new versions of pip have been synced to S3 using: ``` $ pip download --no-cache pip==20.3.4 ... Saved ./pip-20.3.4-py2.py3-none-any.whl Successfully downloaded pip $ pip download --no-cache pip==21.3.1 Collecting pip==21.3.1 ... Saved ./pip-21.3.1-py3-none-any.whl Successfully downloaded pip $ aws s3 sync . s3://heroku-buildpack-python/common/ --exclude "*" --include "*.whl" --dryrun (dryrun) upload: ./pip-20.3.4-py2.py3-none-any.whl to s3://heroku-buildpack-python/common/pip-20.3.4-py2.py3-none-any.whl (dryrun) upload: ./pip-21.3.1-py3-none-any.whl to s3://heroku-buildpack-python/common/pip-21.3.1-py3-none-any.whl $ aws s3 sync . s3://heroku-buildpack-python/common/ --exclude "*" --include "*.whl" upload: ./pip-20.3.4-py2.py3-none-any.whl to s3://heroku-buildpack-python/common/pip-20.3.4-py2.py3-none-any.whl upload: ./pip-21.3.1-py3-none-any.whl to s3://heroku-buildpack-python/common/pip-21.3.1-py3-none-any.whl ``` Closes #1109. GUS-W-8493316.
heroku · Nov 1, 2021 · b82ddba · b82ddba
1 parent c5afa5b
commit b82ddba
Show file tree

Hide file tree

Showing 6 changed files with 48 additions and 54 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,13 +2,16 @@
 
 ## Unreleased
 
+- Update pip from 20.2.4 to: ([#1259](https://github.com/heroku/heroku-buildpack-python/pull/1259))
+  - 20.3.4 for Python 2.7 and 3.5
+  - 21.3.1 for Python 3.6+
 
 ## v201 (2021-10-20)
 
-- Update setuptools from 47.1.1 to: ([#1253](https://github.com/heroku/heroku-buildpack-python/pull/1254))
+- Update setuptools from 47.1.1 to: ([#1254](https://github.com/heroku/heroku-buildpack-python/pull/1254))
   - 50.3.2 for Python 3.5
   - 57.5.0 for Python 3.6+
-- Update wheel from 0.36.2 to 0.37.0 ([#1253](https://github.com/heroku/heroku-buildpack-python/pull/1254)).
+- Update wheel from 0.36.2 to 0.37.0 ([#1254](https://github.com/heroku/heroku-buildpack-python/pull/1254)).
 - Perform editable package `.pth` and `.egg-link` path rewriting at runtime ([#1252](https://github.com/heroku/heroku-buildpack-python/pull/1252)).
 
 ## v200 (2021-10-04)

diff --git a/bin/steps/python b/bin/steps/python
@@ -150,7 +150,7 @@ fi
 
 set -e
 
-PIP_VERSION='20.2.4'
+PIP_VERSION='21.3.1'
 SETUPTOOLS_VERSION='57.5.0'
 WHEEL_VERSION='0.37.0'
 
@@ -160,10 +160,12 @@ if [[ "${PYTHON_VERSION}" == ${PY34}* ]]; then
   SETUPTOOLS_VERSION='43.0.0'
   WHEEL_VERSION='0.33.6'
 elif [[ "${PYTHON_VERSION}" == ${PY27}* || "${PYTHON_VERSION}" == ${PYPY27}* ]]; then
-  # Python 2.7 support was dropped in setuptools 45+.
+  # Python 2.7 support was dropped in pip 21+ and setuptools 45+.
+  PIP_VERSION='20.3.4'
   SETUPTOOLS_VERSION='44.1.1'
 elif [[ "${PYTHON_VERSION}" == ${PY35}* ]]; then
-  # Python 3.5 support was dropped in setuptools 51+.
+  # Python 3.5 support was dropped in pip 21+ and setuptools 51+.
+  PIP_VERSION='20.3.4'
   SETUPTOOLS_VERSION='50.3.2'
 fi
 
@@ -176,7 +178,14 @@ puts-step "Installing pip ${PIP_VERSION}, setuptools ${SETUPTOOLS_VERSION} and w
 #  - we would still have to manage several versions of get-pip.py, to support older Pythons.
 # Instead, we use the pip wheel to install itself, using the method described here:
 # https://github.com/pypa/pip/issues/2351#issuecomment-69994524
-PIP_WHEEL_FILENAME="pip-${PIP_VERSION}-py2.py3-none-any.whl"
+
+PIP_MAJOR_VERSION="${PIP_VERSION%%.*}"
+if [[ "${PIP_MAJOR_VERSION}" -ge 21 ]]; then
+  # Pip 21+ dropped support for Python 2, so no longer uses universal wheels.
+  PIP_WHEEL_FILENAME="pip-${PIP_VERSION}-py3-none-any.whl"
+else
+  PIP_WHEEL_FILENAME="pip-${PIP_VERSION}-py2.py3-none-any.whl"
+fi
 PIP_WHEEL_URL="${S3_BASE_URL}/common/${PIP_WHEEL_FILENAME}"
 PIP_WHEEL="${TMPDIR:-/tmp}/${PIP_WHEEL_FILENAME}"
 

diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb
@@ -22,7 +22,7 @@
           remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION}
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -38,7 +38,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote: -----> Discovering process types
@@ -61,7 +61,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> Requirements file has been changed, clearing cached dependencies
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -94,21 +94,14 @@
     it 'rewrites .pth and .egg-link paths correctly for hooks, later buildpacks, runtime and cached builds' do
       app.deploy do |app|
         expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
-          remote: -----> Installing requirements with pip
-          remote:        Obtaining file:///tmp/build_.*/local_package \\(from -r /tmp/build_.*/requirements.txt \\(line 1\\)\\)
-          remote:        Obtaining gunicorn from git\\+https://github.com/benoitc/gunicorn@20.1.0#egg=gunicorn \\(from -r /tmp/build_.*/requirements.txt \\(line 2\\)\\)
-          remote:          Cloning https://github.com/benoitc/gunicorn \\(to revision 20.1.0\\) to /app/.heroku/src/gunicorn
-          remote:        Installing collected packages: gunicorn, local-package
-          remote:          Running setup.py develop for gunicorn
-          remote:          Running setup.py develop for local-package
-          remote:        Successfully installed gunicorn local-package
+          remote:        Successfully installed gunicorn-20.1.0 local-package-0.0.1
           remote: -----> Running post-compile hook
           remote: ==> .heroku/python/lib/python.*/site-packages/distutils-precedence.pth <==
-          remote: import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get\\(var, 'stdlib'\\) == 'local'; enabled and __import__\\('_distutils_hack'\\).add_shim\\(\\); 
+          remote: .*
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
-          remote: /app/.heroku/src/gunicorn
           remote: /tmp/build_.*/local_package
+          remote: /app/.heroku/src/gunicorn
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/gunicorn.egg-link <==
           remote: /app/.heroku/src/gunicorn
@@ -120,11 +113,11 @@
           remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
           remote: -----> Inline app detected
           remote: ==> .heroku/python/lib/python.*/site-packages/distutils-precedence.pth <==
-          remote: import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get\\(var, 'stdlib'\\) == 'local'; enabled and __import__\\('_distutils_hack'\\).add_shim\\(\\); 
+          remote: .*
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
-          remote: /app/.heroku/src/gunicorn
           remote: /tmp/build_.*/local_package
+          remote: /app/.heroku/src/gunicorn
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/gunicorn.egg-link <==
           remote: /app/.heroku/src/gunicorn
@@ -139,11 +132,11 @@
         # Test rewritten paths work at runtime.
         expect(app.run('bin/test-entrypoints')).to match(Regexp.new(<<~REGEX))
           ==> .heroku/python/lib/python.*/site-packages/distutils-precedence.pth <==
-          import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get\\(var, 'stdlib'\\) == 'local'; enabled and __import__\\('_distutils_hack'\\).add_shim\\(\\); 
+          .*
 
           ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
-          /app/.heroku/src/gunicorn
           /app/local_package
+          /app/.heroku/src/gunicorn
 
           ==> .heroku/python/lib/python.*/site-packages/gunicorn.egg-link <==
           /app/.heroku/src/gunicorn
@@ -155,25 +148,14 @@
           Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
         REGEX
 
-        # Test restoring paths in the cached .pth files works correctly.
+        # Test that the cached .pth files work correctly.
         app.commit!
         app.push!
         expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
-          remote: -----> No change in requirements detected, installing from cache
-          remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
-          remote: -----> Installing SQLite3
-          remote: -----> Installing requirements with pip
-          remote:        Obtaining file:///tmp/build_.*/local_package \\(from -r /tmp/build_.*/requirements.txt \\(line 1\\)\\)
-          remote:        Obtaining gunicorn from git\\+https://github.com/benoitc/gunicorn@20.1.0#egg=gunicorn \\(from -r /tmp/build_.*/requirements.txt \\(line 2\\)\\)
-          remote:          Cloning https://github.com/benoitc/gunicorn \\(to revision 20.1.0\\) to /app/.heroku/src/gunicorn
-          remote:        Installing collected packages: gunicorn, local-package
-          remote:          Running setup.py develop for gunicorn
-          remote:          Running setup.py develop for local-package
-          remote:        Successfully installed gunicorn local-package
+          remote:        Successfully installed gunicorn-20.1.0 local-package-0.0.1
           remote: -----> Running post-compile hook
           remote: ==> .heroku/python/lib/python.*/site-packages/distutils-precedence.pth <==
-          remote: import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get\\(var, 'stdlib'\\) == 'local'; enabled and __import__\\('_distutils_hack'\\).add_shim\\(\\); 
+          remote: .*
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
           remote: /app/.heroku/src/gunicorn
@@ -189,7 +171,7 @@
           remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
           remote: -----> Inline app detected
           remote: ==> .heroku/python/lib/python.*/site-packages/distutils-precedence.pth <==
-          remote: import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get\\(var, 'stdlib'\\) == 'local'; enabled and __import__\\('_distutils_hack'\\).add_shim\\(\\); 
+          remote: .*
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
           remote: /app/.heroku/src/gunicorn

diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb
@@ -11,7 +11,7 @@
         remote: -----> Using Python version specified in Pipfile.lock
         remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
         remote: -----> Installing python-#{python_version}
-        remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+        remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
         remote: -----> Installing dependencies with Pipenv 2020.11.15
         remote:        Installing dependencies from Pipfile.lock \\(.*\\)...
         remote: -----> Installing SQLite3
@@ -33,7 +33,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile...
           remote: -----> Installing SQLite3
@@ -53,7 +53,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock \\(aad8b1\\)...
           remote: -----> Installing SQLite3
@@ -76,7 +76,7 @@
             remote:        Learn More: https://devcenter.heroku.com/articles/python-2-7-eol-faq
             remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
             remote: -----> Installing python-#{LATEST_PYTHON_2_7}
-            remote: -----> Installing pip 20.2.4, setuptools 44.1.1 and wheel 0.37.0
+            remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.0
             remote: -----> Installing dependencies with Pipenv 2020.11.15
             remote:        Installing dependencies from Pipfile.lock \\(b8efa9\\)...
             remote: -----> Installing SQLite3
@@ -153,7 +153,7 @@
           remote:        Learn More: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-3.9.1
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock \\(e13df1\\)...
           remote: -----> Installing SQLite3
@@ -203,7 +203,7 @@
           remote: -----> Using Python version specified in runtime.txt
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{LATEST_PYTHON_3_9}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock \\(75eae0\\)...
           remote: -----> Installing SQLite3
@@ -221,7 +221,7 @@
           remote: -----> Python app detected
           remote: -----> Using Python version specified in Pipfile.lock
           remote: -----> Installing python-#{LATEST_PYTHON_3_9}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock (ef68d1)...
           remote: -----> Installing SQLite3
@@ -241,7 +241,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Your Pipfile.lock \\(aad8b1\\) is out of date. Expected: \\(ef68d1\\).
           remote:        \\[DeployException\\]: .*

diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb
@@ -9,7 +9,7 @@
         remote: -----> Python app detected
         remote: -----> Using Python version specified in runtime.txt
         remote: -----> Installing python-#{python_version}
-        remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+        remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
         remote: -----> Installing SQLite3
         remote: -----> Installing requirements with pip
         remote:        Collecting urllib3
@@ -89,7 +89,7 @@
             remote:  !     Python 2 has reached its community EOL. Upgrade your Python runtime to maintain a secure application as soon as possible.
             remote:        Learn More: https://devcenter.heroku.com/articles/python-2-7-eol-faq
             remote: -----> Installing python-#{LATEST_PYTHON_2_7}
-            remote: -----> Installing pip 20.2.4, setuptools 44.1.1 and wheel 0.37.0
+            remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.0
             remote: -----> Installing SQLite3
             remote: -----> Installing requirements with pip
             remote:        Collecting urllib3
@@ -151,7 +151,7 @@
             remote: -----> Python app detected
             remote: -----> Using Python version specified in runtime.txt
             remote: -----> Installing python-#{LATEST_PYTHON_3_5}
-            remote: -----> Installing pip 20.2.4, setuptools 50.3.2 and wheel 0.37.0
+            remote: -----> Installing pip 20.3.4, setuptools 50.3.2 and wheel 0.37.0
             remote: -----> Installing SQLite3
             remote: -----> Installing requirements with pip
             remote:        Collecting urllib3
@@ -208,7 +208,7 @@
           remote: -----> Python app detected
           remote: -----> Using Python version specified in runtime.txt
           remote: -----> Installing pypy2.7-#{LATEST_PYPY_2_7}
-          remote: -----> Installing pip 20.2.4, setuptools 44.1.1 and wheel 0.37.0
+          remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -227,7 +227,7 @@
           remote: -----> Python app detected
           remote: -----> Using Python version specified in runtime.txt
           remote: -----> Installing pypy3.6-#{LATEST_PYPY_3_6}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -270,7 +270,7 @@
           remote: -----> Python version has changed from python-#{LATEST_PYTHON_3_6} to python-#{LATEST_PYTHON_3_9}, clearing cache
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Installing python-#{LATEST_PYTHON_3_9}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3

diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb
@@ -29,7 +29,7 @@
           remote: -----> Stack has changed from heroku-18 to heroku-20, clearing cache
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Installing python-3.6.12
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -58,7 +58,7 @@
           remote: -----> Stack has changed from heroku-20 to heroku-18, clearing cache
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 20.2.4, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3