Python3: Impossible to update pip any time a pip update is released

[gatto]

Possibly similar to #57612, Homebrew/legacy-homebrew#26900, https://discourse.brew.sh/t/homebrews-pip3-shortcut-via-pip-has-hard-coded-version-dependency-causing-errors/4912.

• [ • ] ran brew update and can still reproduce the problem?
• [ • ] ran brew doctor, fixed all issues and can still reproduce the problem?
• [ • ] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link? (sorry, I don't know how this works)

What you were trying to do (and why)

I was trying, in my fully upgraded brew python installation, to upgrade pip from 20.1.1 which is installed with python, to the newest 20.2.2 which includes some important changes.

What happened (include command output)

Pip crashes and won't ever work again unless I reinstall the formula.

Command output

fabio@gattoBook ~ % python3 -m pip install --upgrade pip

Collecting pip

Downloading pip-20.2.2-py2.py3-none-any.whl (1.5 MB)

|████████████████████████████████| 1.5 MB 488 kB/s

Installing collected packages: pip

Attempting uninstall: pip

Found existing installation: pip 20.1.1

Uninstalling pip-20.1.1:

Successfully uninstalled pip-20.1.1

Successfully installed pip-20.2.2

fabio@gattoBook ~ % pip3 list

Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 583, in _build_master

ws.require(requires)

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 900, in require

needed = self.resolve(parse_requirements(requirements))

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 791, in resolve

raise VersionConflict(dist, req).with_context(dependent_req)

pip._vendor.pkg_resources.VersionConflict: (pip 20.2.2 (/usr/local/lib/python3.8/site-packages), Requirement.parse('pip==20.1.1'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "/usr/local/opt/python@3.8/bin/pip3", line 33, in 

sys.exit(load_entry_point('pip==20.1.1', 'console_scripts', 'pip3')())

File "/usr/local/opt/python@3.8/bin/pip3", line 25, in importlib_load_entry_point

return next(matches).load()

File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/metadata.py", line 77, in load

module = import_module(match.group('module'))

File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/init.py", line 127, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "", line 1014, in _gcd_import

File "", line 991, in _find_and_load

File "", line 975, in _find_and_load_unlocked

File "", line 671, in _load_unlocked

File "", line 783, in exec_module

File "", line 219, in _call_with_frames_removed

File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 10, in 

from pip._internal.cli.autocompletion import autocomplete

File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in 

from pip._internal.cli.main_parser import create_main_parser

File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in 

from pip._internal.cli import cmdoptions

File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in 

from pip._internal.cli.progress_bars import BAR_TYPES

File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/progress_bars.py", line 12, in 

from pip._internal.utils.logging import get_indentation

File "/usr/local/lib/python3.8/site-packages/pip/_internal/utils/logging.py", line 18, in 

from pip._internal.utils.misc import ensure_dir

File "/usr/local/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 21, in 

from pip._vendor import pkg_resources

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 3252, in 

def _initialize_master_working_set():

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 3235, in _call_aside

f(*args, **kwargs)

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 3264, in _initialize_master_working_set

working_set = WorkingSet._build_master()

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 585, in _build_master

return cls._build_from_requirements(requires)

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 598, in _build_from_requirements

dists = ws.resolve(reqs, Environment())

File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pkg_resources/init.py", line 786, in resolve

raise DistributionNotFound(req, requirers)

pip._vendor.pkg_resources.DistributionNotFound: The 'pip==20.1.1' distribution was not found and is required by the application

fabio@gattoBook ~ %
  

What you expected to happen

Pip not to crash.

Step-by-step reproduction instructions (by running brew install commands)

brew install python3
pip3 list --outdated
pip3 install -U pip
pip3 list

[jnozsc]

not an official solution, just a workaround

modify /usr/local/bin/pip3

to

#!/usr/local/opt/python@3.8/bin/python3.8
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

the issue is likely caused by easy_install, which is deprecated for a while. I guess Homebrew keeps it for back compatibility

[gatto]

@jnozsc Could we include this code in the formula? So that the problem is solved for everyone instead of just me

[ghost]

I was struggling with this and thought it was just me. Installing the updates to setuptools and pip after installing python via brew causes all kinds of grief. Aliases are replaced with binaries in /usr/local/bin and there is an error that python requires an older version of pip (the one that was installed when brew installed python (currently 20.1.1).

For now at least notate the python page that updating pip and setuptools is broken or inform users of the correct way to update those tools.

my question with output on discourse.brew

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[gatto]

I agree that the issue is stale. However, it's also still present…

[emandret]

Still stuck with it...

[ghost]

@jnozsc Could we include this code in the formula? So that the problem is solved for everyone instead of just me

Did this actually solve the issue for you?

[gatto]

@jnozsc Could we include this code in the formula? So that the problem is solved for everyone instead of just me

Did this actually solve the issue for you?

Not really, I don't know how to use it.

[ghost]

@gatto In that case please don't give me a thumbs down. LOL

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[pradyunsg]

FYI: pip 20.3 has been released, and Homebrew users are going to hit this again.

pypa/pip#9179

[pradyunsg]

homebrew-core/Formula/python@3.9.rb

Line 235 in aaee906

system bin/"python3", "-s", "setup.py", "--no-user-cfg", "install",

This is happening since Homebrew is using easy_install to generate the scripts. This is the cause of the failures.

[cbrnr]

@pradyunsg so what would the solution be to prevent this issue? What's the alternative to using easy_install?

[pradyunsg]

Using get-pip.py (use the tags on the GitHub repo to get exact versions: https://github.com/pypa/get-pip/), which would do the script-generation properly.

[alebcay]

Using get-pip.py (use the tags on the GitHub repo to get exact versions: pypa/get-pip), which would do the script-generation properly.

Just tried playing around with this a bit today. One concern that I have is that we have no way to build pip, setuptools, and/or wheel (can't run get-pip.py with --no-binary :all: because we need setuptools to build stuff).

One option (maybe not the best) is to get some kind of portable Python installation or venv to bootstrap this build (with the binary/wheels of pip, setuptools, wheel) so that we can have a source-built pip, setuptools, and wheel in the "real" Python distribution.

[cbrnr]

I found that only /usr/local/opt/python@3.9/bin/pip3 contains the version check, but not /usr/local/bin/pip3. Not sure if both versions get installed or if the latter gets created after a manual pip upgrade, but if I do not add /usr/local/opt/python@3.9/libexec/bin to my PATH (which doesn't happen by default, and symlinks in there point to /usr/local/opt/python@3.9/bin), everything works as expected.

[pradyunsg]

FWIW, modern CPython also ships with ensurepip.

[rgoldberg]

pip has broken multiple times in the last few weeks for me due to the python-formula-generated pip script required pip version lagging behind pip updates. This must have happened to many others.

How hard will it be for someone with the requisite knowledge to rewrite the install script to solve this issue by using get-pip.py?

I don't know Python, its install routines, homebrew internals, or Ruby, so it would be inefficient for me to create a PR myself.

[SMillerDev]

I don't know Python, its install routines, homebrew internals, or Ruby, so it would be inefficient for me to create a PR myself.

And this is exactly why we don't have a fix yet. As far as I'm aware no person exists that matches all these criteria.

[carlocab]

To be fair I think the required level of understanding of Homebrew internals and Ruby is quite low. The real barrier is someone who knows the details of installing Python and has the time to put into this. (I'm assuming the way that Homebrew installs Python isn't so idiosyncratic as to not be too difficult to understand for said person.)

[rgoldberg]

Can people who have some of the requisite knowledge & who have free time post what they know, and what they'd need help with?

Maybe we can cobble together all the expertise necessary from a few people working together.

If people can fill in gaps in each other's knowledge, the problem can be tackled faster than one person having to investigate by themselves everything that they don't know.

[pradyunsg]

Just tried playing around with this a bit today. One concern that I have is that we have no way to build pip, setuptools, and/or wheel (can't run get-pip.py with --no-binary :all: because we need setuptools to build stuff).

Wait, why would you want to run with --no-binary? You're disabling the use of wheel files, which are basically zip files containing only the source code and no compiled artifacts.

[alebcay]

Wait, why would you want to run with --no-binary? You're disabling the use of wheel files, which are basically zip files containing only the source code and no compiled artifacts.

Perhaps I am misunderstanding what wheels are, but my understanding is that wheels are compiled artifacts (hence why on PyPi when you download a package, there may be many different wheel files - for different Python version/OS combinations), and that specifying --no-binary :all: instructs pip to rely only on source distributions, not compiled artifacts.

Please correct me if I'm wrong.

[rgoldberg]

Why is --without-ensurepip used by the python@3.9 formula?

Would removing that, and removing the usage of easy_install, fix things?

Or must get-pip.py or something else be explicitly used?

[pradyunsg]

wheels are compiled artifacts

Wheels are zip files containing the final files that go into the site-packages (which is on Python's import path, so import xyz works after unpacking the wheel correctly).

These files can be compiled artifacts, but when the compatibility tag is "none-any", it's a pure-Python wheel -- only .py files, no .so or .pyd. pip, setuptools, and wheel have pure-Python wheels. Unless you wanna pull a "zip files with pure-Python code are still opaque compiled artifacts" as certain downstream publishers have (for IMO no benefits and more pain for themselves), don't use --only-binary :all:.

Why is --without-ensurepip used by the python@3.9 formula?

🤷

FWIW, this feels like a consequence of deciding that "wheels are opaque compiled artifacts" and deciding to not use them. I'd strongly suggest avoiding this as well.

[rgoldberg]

@pradyunsg Thanks for the info. Under normal circumstances, would removing the use of --without-ensurepip & easy_install properly install the latest versions of pip, setuptools & wheel without hard-coding any package versions in scripts?

If so, then all we need to do is find out why Homebrew doesn't do that, find workarounds for whatever issues are noted, then rip out & replace the existing odd install options / procedure.

[rgoldberg]

--without-ensurepip was added to python@3.7 in commit 9d037c1 from PR Homebrew/legacy-homebrew#41544 by @tdsmith for issue Linuxbrew/legacy-linuxbrew#469.

--without-ensurepip has been in python@3.8 & python@3.9 since their initial commits.

The PR says that ensurepip installs an old version of pip.

Why would ensurepip install an old version of pip?

Is there any way of forcing ensurepip to install the latest version, or an explicitly specified version that can be manually updated in the formula on every pip upgrade?

In any of the cases, does ensurepip hard-code any package versions like the current formula does for pip?

If so, then using ensurepip won't solve this problem, unless it somehow ensures that future updates to any packages whose version get hard-coded automatically update the hard-coded versions (which I doubt would occur, but, hey, it's theoretically possible…).

Are there any other reasons why ensurepip isn't used?

Who is the person currently in charge of the python formulae? Can they be roped into this discussion for guidance?

[alebcay]

These files can be compiled artifacts, but when the compatibility tag is "none-any", it's a pure-Python wheel -- only .py files, no .so or .pyd. pip, setuptools, and wheel have pure-Python wheels. Unless you wanna pull a "zip files with pure-Python code are still opaque compiled artifacts" as certain downstream publishers have (for IMO no benefits and more pain for themselves), don't use --only-binary :all:.

Thanks for the information, that does help clear things up a bit. Homebrew has a penchant for preferring to work from source distributions (no compiled artifacts), so we have been using --only-binary :all: (for more info on this Homebrew policy, feel free to read https://docs.brew.sh/Acceptable-Formulae#we-dont-like-binary-formulae at your leisure).

The fact that pip, setuptools, and wheel have pure-Python wheels is thus news to me. I can try fiddling around a bit more with get-pip.py with that information in mind (without --only-binary :all:).

---

Who is the person currently in charge of the python formulae? Can they be roped into this discussion for guidance?

Homebrew differs from most other package managers out there in that there is no particular maintainer responsible for a particular formula (working off of a back-of-the-napkin calculation, with ~5382 formulae at the time of writing and 26 maintainers currently, we'd each have to manage over 200 formulae, which is not sustainable).

That's why we rely so heavily on community contributions, especially from people that actually know what they're doing and what they're talking about, because frankly, most of us Homebrew maintainers don't know all the ins and outs of Python distribution. This community-driven model is beneficial in that pretty much anyone can get involved, but at times like this, we can also see how things come to an impasse of sorts.

As such, even though this issue is still ongoing, I'd like to take a moment to thank both of you for your input so far, @pradyunsg and @rgoldberg, for providing feedback, answering questions, and proposing some solutions to explore as well. ✨

[rgoldberg]

@alebcay Thanks for the info and for the thanks. I wish I knew more about the associated technologies so I could provide answers instead of asking questions. If anyone knows anyone who might have answers, please mention them in this issue so we can canvass their input.

Thanks everyone for your input.

[iMichka]

The PR says that ensurepip installs an old version of pip.
Why would ensurepip install an old version of pip?

Because ensure-pip is provided by the python source files: it installs the pip version that is hard coded at the moment of delivery of a new Python version. So pip will continue to evolve and drift away from that version.

But I guess we could do a pip update during the build to get the latest version? It does not look really deterministic though as we might pick up any random latest pip version.

On the other side, when reading this, I am unsure get-pip.py is a good solution either: pypa/get-pip#60

[rgoldberg]

But I guess we could do a pip update during the build to get the latest version? It does not look really deterministic though as we might pick up any random latest pip version.

It looks like you can upgrade pip packages to specific versions using the following syntax:

pip install --upgrade $(package)==$(version)

So the formula can update specified packages to specified versions.

As long as this generates scripts that don't hard-code package versions, and as long as there aren't any other problems, it looks like we can solve this issue by:

• removing --without-ensurepip
• removing easy_install & associated code / files
• making install update pip, wheel, setuptools, and/or other packages to versions specified in the formula
• updating the specified package versions whenever new (or just other) versions are determined to be better than the existing specified versions

We wouldn't need to use get-pip.py.

Does anyone see problems with this approach? There could be many, as I don't know any of the involved technologies…

Would the normal Python install still include easy install, or would / should it be omitted if the formula no longer uses it?

[tdsmith]

I can't repro this and I'm not sure how people are hitting it. I don't see a plausible root cause.

This is happening since Homebrew is using easy_install to generate the scripts. This is the cause of the failures.

We aren't -- invoking setup.py isn't the same as using easy_install. This is just an installation from sdist. pip does this in the absence of a wheel.

The presenting complaint sounds like an instance of a pip script in $PATH that doesn't match the version of pip discovered in sys.path. brew post_install python should correct a number of these cases.

[tdsmith]

brew post_install python should correct a number of these cases.

Although, on reflection, this would downgrade pip, so this probably isn't what people are looking for :)

I'm curious whether python -m pip works for affected users.

[pradyunsg]

I'm curious whether python -m pip works for affected users.

It will, yea.

The issue here is that the pip executable generated by easy_install is... "bad". Using python -m pip is a good workaround -- python -m pip install pip --force-reinstall would likely be a good way to regenerate all the things with pip itself. :)

[tdsmith]

Oh, sorry Pradyun, I shouldn't be lecturing you on packaging.

Why isn't pip rewriting this file when pip upgrades pip?

[pradyunsg]

easy_install puts it in a weird place, that probably shows up earlier in the PATH, and doesn't note that as "I installed a thing here"... maybe?

Honestly, I don't know much about about easy_install's behavior beyond that it shouldn't be used because it's horrible in more ways than one, and I have no interest in learning about how. I don't really care about the nuances here -- don't use easy_install is an easier user-facing message anyway. 🤷🏽

We aren't -- invoking setup.py isn't the same as using easy_install.

Sure -- setup.py install and easy_install are different "things" to invoke on the CLI. It doesn't matter though, coz they are both internally invoking the same code in setuptools, which has not been touched because, honestly, no one should be depending on it and should use pip.

---

I'm unsubscribing from this issue right now, because I don't want to keep repeating myself here. My final notes here are: I think it'd be better if Brew's Python formulae stop going out of the way to disable all the supported mechanisms for distributing pip and intentionally use a mechanism for install that's (a) deprecated and (b) known to cause issues now. The supported installation methods are listed on https://pip.pypa.io/en/stable/installing/ -- pick one from there and use that please.

[pradyunsg]

I'm unsubscribing from this issue right now, because I don't want to keep repeating myself here.

To be clear, this isn't because I don't think this isn't a problem worth fixing (it is, and even I have hit it in the past) or because of something that someone else has said here/elsewhere or whatever. Rather, this is a matter of me prioritizing my time since I think everything that I had to say here has been said.

Feel free to @ mention me if someone wants me to review the PR that fixes this, and I'll be more than happy to provide inputs on that. :)

[tdsmith]

Thanks for your help and patience, Pradyun.

I'll drop off of this ticket too -- I'm retired from Homebrew :) and I don't think humans should use these packages; you should use pyenv instead -- but I think a useful thing that Homebrew could do here would be:

• use ensurepip while installing Python (accepting that this will temporarily install what is probably a stale version of pip)
• use the pip that ensurepip installed to install the resourced pip

I think we didn't do this originally just because it felt redundant. It is :) but it's not expensive.

Reevaluating Homebrew's general approach in the light of continued progress in the Python packaging ecosystem could make sense. Using pip to install sdists instead of invoking setup.py directly could be productive.

[alebcay]

#67030 has been merged just now, which should fix the pip version being hardcoded into the pip script. Please holler if the issue persists and we can reopen this and/or investigate further.

Some other ideas for bringing the Homebrew installation process of Python more in line with modern best practices were also spurred over the course of this discussion, but considering the length and scope of this issue, it may be better to discuss other improvements/problems in new issues/PRs.