list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version' #5429

daavve · 2018-05-20T04:19:37Z

Environment

pip version:
Python version:
OS:

Arch Linux, Testing version of pip: https://www.archlinux.org/packages/testing/any/python-pip/
Description

$ pip list -o

Exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
status = self.run(options, args)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 145, in run
packages = self.get_outdated(packages, options)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 156, in get_outdated
dist for dist in self.iter_packages_latest_infos(packages, options)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 157, in
if dist.latest_version > dist.parsed_version
TypeError: '>' not supported between instances of 'Version' and 'Version'

daavve · 2018-05-20T04:22:54Z

Reported downstream: https://bugs.archlinux.org/index.php?do=details&task_id=58670

yan12125 · 2018-05-21T13:55:01Z

Looks like an issue related to deblundling. If I modify the get_outdated() function in /usr/lib/python3.6/site-packages/pip/_internal/commands/list.py to:

    def get_outdated(self, packages, options):
        ret = []
        for dist in self.iter_packages_latest_infos(packages, options):
            print(dist.latest_version.__class__)
            print(dist.parsed_version.__class__)
            if dist.latest_version > dist.parsed_version:
                ret.append(dist)
        return ret

pip list -o prints:

<class 'pip._vendor.packaging.version.Version'>
<class 'pkg_resources.extern.packaging.version.Version'>
Exception:
Traceback (most recent call last):                                                                                                      
  File "/usr/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main                                               
    status = self.run(options, args)                                                                                                    
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 145, in run                                              
    packages = self.get_outdated(packages, options)                                                                                     
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 159, in get_outdated                                     
    if dist.latest_version > dist.parsed_version:                                                                                       
TypeError: '>' not supported between instances of 'Version' and 'Version'

pip._vendor.packaging.version.Version and pkg_resources.extern.packaging.version.Version are actually the same class (both are packaging.version.Version), while Python does not think so.

pradyunsg · 2018-05-21T14:14:54Z

Thanks for looking into this @yan12125! :)

Indeed, this is a downstream packaging issue.

pradyunsg · 2018-05-21T14:16:46Z

Adding to my above comment, I don't see anything actionable on this from pip's on this -- it's due to how Arch Linux is debundling pip.

daavve · 2018-05-21T14:53:47Z

I think you are right. This looks like an Arch issue.

yan12125 · 2018-05-21T16:23:12Z

I managed to patch pkg_resources locally to make pip list -o work with Arch Linux packages. Hopefully that helps other distributions using debundling. This patch is created against python-setuptools 1:39.2.0-1.

--- pkg_resources/__init__.py.orig	2018-05-22 00:21:05.592049050 +0800
+++ pkg_resources/__init__.py	2018-05-22 00:21:19.378315213 +0800
@@ -71,11 +71,11 @@
 
 from . import py31compat
 from pkg_resources.extern import appdirs
-from pkg_resources.extern import packaging
-__import__('pkg_resources.extern.packaging.version')
-__import__('pkg_resources.extern.packaging.specifiers')
-__import__('pkg_resources.extern.packaging.requirements')
-__import__('pkg_resources.extern.packaging.markers')
+import packaging
+__import__('packaging.version')
+__import__('packaging.specifiers')
+__import__('packaging.requirements')
+__import__('packaging.markers')
 
 
 if (3, 0) < sys.version_info < (3, 3):

eli-schwartz · 2018-05-22T05:22:24Z

Adding to my above comment, I don't see anything actionable on this from pip's on this -- it's due to how Arch Linux is debundling pip.

We're debundling according to the method nominally supported by pip itself, so I think it should be actionable here. :)

pip._internal.index.PackageFinder attempts to use pip._vendor.packaging.version.parse (which is dist.latest_version in the traceback), then compares it to pip._internal.utils.misc.get_installed_distributions which uses pip._vendor.pkg_resources to return a set of pkg_resources packages -- even though they're the same files, they appear in different locations on the import path, and python complains.

This somehow works for pip when using vendored everything, but only because the vendored pkg_resources is then additionally modified from stock pkg_resources to import everything from the top-level _vendor.packaging

...

I don't fully grok the magic used by pip's vendored() and pkg_resource's VendorImporter(), but it looks like they might be incompatible, and pip is papering over this logic by completely ignoring the latter in its own vendoring code.

pradyunsg · 2018-05-22T05:41:01Z

Hey @eli-schwartz!

Could you take a look at #5346 (comment) and provide information about the people at Arch Linux to ping for such things? :)

(I can't respond to your comment now because I'm super short on time; I'll reopen this issue so that I remember that I need to get back to this.)

yan12125 · 2018-05-22T17:13:51Z

I created a more general patch:

--- pkg_resources/extern/__init__.py.orig	2018-05-23 00:58:31.680640763 +0800
+++ pkg_resources/extern/__init__.py	2018-05-23 00:58:47.678203875 +0800
@@ -48,7 +48,7 @@
                 # on later Python versions to cause relative imports
                 # in the vendor package to resolve the same modules
                 # as those going through this importer.
-                if sys.version_info > (3, 3):
+                if sys.version_info > (3, 3) and prefix != '':
                     del sys.modules[extant]
                 return mod
             except ImportError:

Anyway this looks like a setuptools issue rather than a pip one.

eli-schwartz · 2018-05-23T00:37:02Z

@jaraco, can you confirm?

yan12125 · 2019-01-05T03:12:40Z

The issue reappeared with pip git-master and Python 3.7.2 even with the setuptools patch in #5429 (comment). This time the fix should go to pip IMO. I'm currently working on this.

valtron · 2019-02-04T06:09:34Z

Reinstall pip to fix this:

pip3 install --ignore-installed pip

…ces (#6113) With the original `vendored()` implementation and such an initialization sequence: ``` vendored("packaging") vendored("packaging.version") ``` In `sys.modules`, `pip._vendor.packaging` is correctly connected to the debundled `packaging`, while `pip._vendor.packaging.version` is not, as the latter is `__import__`ed from the existing `pip._vendor.packaging` module. That results in the same issue as #5429 - `pip._vendor.packaging.version.Version` and `packaging.version.Version` cannot be compared. This patch attempts to fix this issue by skipping `__import__` from the vendored name. This is safe because `vendored()` is called only when `DEBUNDLED = True`, and vendored libraries are already deleted as per [debundling instructions](https://github.com/pypa/pip/blob/master/src/pip/_vendor/README.rst#debundling).

ntninja · 2019-05-12T15:36:14Z

I'm affected as well by this on Debian unstable and neither of @yan12125's patches worked. 🙁

yan12125 · 2019-05-12T16:03:07Z

The issue still exists in pip 19.1?

eli-schwartz · 2019-05-12T16:06:06Z

Debian unstable still packages pip 18.1, so @Alexander255's issue is definitely not confirmed with pip 19.

bernd-wechner · 2019-07-11T05:39:24Z

pip3 install --ignore-installed pip

Wow I just did that because I have this issue on Mint 19. Only it broke pip. Awesome.

$ pip3 install --ignore-installed pip
Collecting pip
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 814kB/s 
Installing collected packages: pip
Successfully installed pip-19.1.1
$ pip3
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

Any pointer appreciated. That sux!

Addendum: fixed it my uninstalling pip3 with apt and reinstalling.

bernd-wechner · 2019-07-11T06:00:04Z

@yan12125

and prefix != '':

I made that patch and still had the problem. So undid it.

eli-schwartz · 2019-07-11T06:09:05Z

Wow I just did that because I have this issue on Mint 19. Only it broke pip. Awesome.

Don't follow bad advice from people who don't know what they are talking about and don't have context on the issue. Speaking with my "I am a developer for a linux distro" hat on, do not ever sudo pip anything. Ever. And definitely, definitely, definitely, do not try to install pip from your distro package manager and then update it using pip, because you WILL break it and as extensively discussed in #5346, and ideally pip would learn to say "you are not allowed to update this using pip, please use your package manager instead" as tracked by #5605.

You will need to either install pip via apt, or install pip via get-pip.py, and make sure you don't have bits and pieces of the wrong thing in your PYTHONPATH.

...

As for why Linux Mint's pip package has this bug, you need to discuss this with the Debian packagers of python3-pip. My guess is they are shipping a broken setuptools. See https://bugs.archlinux.org/task/58670 for more details.

bernd-wechner · 2019-07-11T07:02:02Z

Thanks enormously for informative advice. I uninstalled the Debian pip packages and installed get-pip.py as per the Python website and now pip works fine, and I have pip installing python3 packages (whereas Debian packages install pip2 and pip3 and link pip to pip2 still - and I like many, have near zero interest in Python 2 anymore).

And pip list -o works fine now too. Basically everything seems fine after removing the debian packages and using get-pip.py.

I do have one question though.

do not ever sudo pip anything.

I mildly agree with the principle here, but what if I want my packages installed in /usr/lib/python3.6/dist-packages say and not in my home dir? In fact with this pip I just installed it tries that and bombs suggesting I use the --user option. What is your recommendation in that space?

pradyunsg · 2019-07-11T19:10:12Z

what if I want my packages installed in /usr/lib/python3.6/dist-packages

What's the use case for not using a Debian-provided pip but still wanting to install to a location where that would install packages?

pradyunsg · 2019-07-11T19:11:34Z

What is your recommendation in that space?

As the fellow who added that to message -- I'd say use --user as the message suggests. 🙃

bernd-wechner · 2019-07-11T23:10:50Z

what if I want my packages installed in /usr/lib/python3.6/dist-packages

What's the use case for not using a Debian-provided pip but still wanting to install to a location where that would install packages?

Forgive me, but the obvious: to make the packages available not to the current user and Python code they run, but to other users, including system services that need access to them. The use case is not to install them where the Debian packaged pip would install them, the use case is system wide installation (vs. local user installation).

With all due respect, that is such a common and ordinary and fairly obvious use case I'd have thought that pip by default installs there (I have to use --user flag to get around it) and I forgot to state it explicitly.

eli-schwartz · 2019-07-11T23:16:07Z

You're installing system services without using the package manager? Personally I'd avoid that sort of thing. Then again, Arch Linux does make it very easy to empower the masses to create system packages.

...

Using sudo pip means that pip can and will overwrite files tracked by the package manager. Do you have software on your system which is written in python and distributed via apt? Python is a popular programming language for desktop software, there are DEs written in python. Do you want to break that software by overwriting system packages with incompatible components?

pradyunsg · 2019-07-12T04:40:44Z

With all due respect, that is such a common and ordinary and fairly obvious use case I'd have thought that pip by default installs there (I have to use --user flag to get around it) and I forgot to state it explicitly.

Yep yep - #1668. It's a bit tricker than it looks on the surface. 🙃

bernd-wechner · 2019-07-12T07:30:40Z

Using sudo pip means that pip can and will overwrite files tracked by the package manager. Do you have software on your system which is written in python and distributed via apt? Python is a popular programming language for desktop software, there are DEs written in python. Do you want to break that software by overwriting system packages with incompatible components?

Guess you're right. Never really thought about it. Still that does create a conundrum that is discussed in the issue predyunsg mentioned via a linked to item... namely that the OS package (Ubuntu apt repos in my case) are generally way out of date ... and so if you want the latest at a system level it's pip or conda I guess.

Of course I guess I concur, to be safe just install to home dir as a first cut.

In the end though I admit I work on a few premises:

That anybody installing python packages (apt, pip, conda?) also takes the trouble to ensure the python path includes the directory they use to install them.
That python packages on the whole don't do backward breaks, i.e. maintain backward compatibility as a priority (inside of a major Python release in any case).

Of course as you point out those premises may be a little naive, and introduce unnecessary risk if taken as truths and installing with pip to system dirs.

it does leave the question open (as per predyunsg's linked item) why on earth pip defaults to system dirs, if the wisdom you share is widely held in the community of people who develop and maintain pip! It does indeed seem sensible to default to local.

pradyunsg · 2019-07-12T07:42:40Z

why on earth pip defaults to system dirs

Because... no one's put in the effort to change that and it's a non-trivial change. :)

martin3000 · 2019-07-28T17:31:35Z

In list.py I changed

    def get_outdated(self, packages, options):
        return [
            dist for dist in self.iter_packages_latest_infos(packages, options)
            if dist.latest_version > dist.parsed_version
        ]

to

    def get_outdated(self, packages, options):
        return [
            dist for dist in self.iter_packages_latest_infos(packages, options)
            if str(dist.latest_version) > str(dist.parsed_version)
        ]

and it works.

yan12125 · 2019-07-29T02:04:40Z

@martin3000 That patch may lead to incorrect results as PEP 440 version comparisons are not plain string comparisons.

@pradyunsg Could you consider closing this issue to send a message that "this has been fixed"? Specifically, users need setuptools >= 40.0.0 and pip >= 19.1. I don't think this ticket needs more discussions about patches for pip/setuptools unless pip list -o would be broken again.

martin3000 · 2019-07-29T07:03:25Z

Comparing the strings ist not ok, but here in Ubuntu 19.04 it was crashing.

yan12125 · 2019-07-29T12:06:43Z

Apparently Ubuntu 19.04 ships with pip 18.1 [1]. The issue is fixed since pip 19.1 (commit 8ef3283, specifically). You may want to report the issue to Ubuntu or Debian developers, so that they can fix their package by either upgrading pip or backporting the relevant commit.

Alternatively, you can also uninstall the system pip package with apt and install the latest pip following https://pip.pypa.io/en/stable/installing/. Note that it's better to keep only one of Ubuntu's pip and the one from pypa.io, or there might be errors like #5429 (comment).

[1] https://launchpad.net/ubuntu/disco/+source/python-pip

jeremy-rifkin · 2019-08-31T15:50:05Z

Had this issue with ubuntu 18 and apt's python3-pip package (pip 9.0.1).
Fixed by uninstalling pip 9.01 and using get-pip.py to install pip 19.2.

pradyunsg · 2019-12-18T04:56:40Z

Could you consider closing this issue to send a message that "this has been fixed"? Specifically, users need setuptools >= 40.0.0 and pip >= 19.1.

Sounds reasonable!

verhovsky · 2020-01-02T16:02:56Z

This might be unsafe, but

sudo -H pip3 install --upgrade pip

Solved it for me on Ubuntu 18.04. Make sure you do sudo, otherwise you'll get ModuleNotFoundError: No module named 'pip'.

pradyunsg added the project: <downstream> When the cause/effect is related to redistributors label May 21, 2018

pradyunsg added the S: needs triage Issues/PRs that need to be triaged label May 21, 2018

daavve closed this as completed May 21, 2018

pradyunsg added resolution: no action When the resolution is to not do anything and removed S: needs triage Issues/PRs that need to be triaged labels May 21, 2018

pradyunsg reopened this May 22, 2018

pradyunsg removed the resolution: no action When the resolution is to not do anything label May 22, 2018

eli-schwartz mentioned this issue May 27, 2018

vendoring logic subtly breaks later usage of unvendored system versions after importing pkg_resources.extern pypa/setuptools#1383

Open

FriedrichFroebel mentioned this issue Jul 15, 2018

TypeError with pip 10.0.1 jgonggrijp/pip-review#63

Closed

This was referenced Jan 4, 2019

Fix handling of requests exceptions when dependencies are debundled #6111

Closed

Error in --outdated and --uptodate parameters from pip list #6012

Closed

yan12125 mentioned this issue Jan 5, 2019

Ensure all pip._vendor.* modules are mapped to debundled correspondences #6113

Merged

woky mentioned this issue Sep 24, 2019

ModuleNotFoundError: No module named 'pkg_resources.extern' sarugaku/requirementslib#185

Closed

bartTC mentioned this issue Nov 8, 2019

HTTPError and TypeError with pip-check installed by pip3 bartTC/pip-check#23

Closed

pradyunsg closed this as completed Dec 18, 2019

tibor mentioned this issue Jan 13, 2020

Windows subsystem: Ubuntu error when "pipupgrade --latest --yes" achillesrasquinha/pipupgrade#34

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Feb 1, 2020

lock bot locked as resolved and limited conversation to collaborators Feb 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version' #5429

list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version' #5429

daavve commented May 20, 2018

daavve commented May 20, 2018 •

edited

Loading

yan12125 commented May 21, 2018

pradyunsg commented May 21, 2018

pradyunsg commented May 21, 2018

daavve commented May 21, 2018

yan12125 commented May 21, 2018

eli-schwartz commented May 22, 2018 •

edited

Loading

pradyunsg commented May 22, 2018

yan12125 commented May 22, 2018

eli-schwartz commented May 23, 2018

yan12125 commented Jan 5, 2019

valtron commented Feb 4, 2019

ntninja commented May 12, 2019

yan12125 commented May 12, 2019

eli-schwartz commented May 12, 2019

bernd-wechner commented Jul 11, 2019 •

edited

Loading

bernd-wechner commented Jul 11, 2019 •

edited

Loading

eli-schwartz commented Jul 11, 2019

bernd-wechner commented Jul 11, 2019

pradyunsg commented Jul 11, 2019

pradyunsg commented Jul 11, 2019

bernd-wechner commented Jul 11, 2019

eli-schwartz commented Jul 11, 2019

pradyunsg commented Jul 12, 2019

bernd-wechner commented Jul 12, 2019

pradyunsg commented Jul 12, 2019

martin3000 commented Jul 28, 2019 •

edited

Loading

yan12125 commented Jul 29, 2019

martin3000 commented Jul 29, 2019

yan12125 commented Jul 29, 2019

jeremy-rifkin commented Aug 31, 2019

pradyunsg commented Dec 18, 2019

verhovsky commented Jan 2, 2020 •

edited

Loading

list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version' #5429

list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version' #5429

Comments

daavve commented May 20, 2018

daavve commented May 20, 2018 • edited Loading

yan12125 commented May 21, 2018

pradyunsg commented May 21, 2018

pradyunsg commented May 21, 2018

daavve commented May 21, 2018

yan12125 commented May 21, 2018

eli-schwartz commented May 22, 2018 • edited Loading

pradyunsg commented May 22, 2018

yan12125 commented May 22, 2018

eli-schwartz commented May 23, 2018

yan12125 commented Jan 5, 2019

valtron commented Feb 4, 2019

ntninja commented May 12, 2019

yan12125 commented May 12, 2019

eli-schwartz commented May 12, 2019

bernd-wechner commented Jul 11, 2019 • edited Loading

bernd-wechner commented Jul 11, 2019 • edited Loading

eli-schwartz commented Jul 11, 2019

bernd-wechner commented Jul 11, 2019

pradyunsg commented Jul 11, 2019

pradyunsg commented Jul 11, 2019

bernd-wechner commented Jul 11, 2019

eli-schwartz commented Jul 11, 2019

pradyunsg commented Jul 12, 2019

bernd-wechner commented Jul 12, 2019

pradyunsg commented Jul 12, 2019

martin3000 commented Jul 28, 2019 • edited Loading

yan12125 commented Jul 29, 2019

martin3000 commented Jul 29, 2019

yan12125 commented Jul 29, 2019

jeremy-rifkin commented Aug 31, 2019

pradyunsg commented Dec 18, 2019

verhovsky commented Jan 2, 2020 • edited Loading

daavve commented May 20, 2018 •

edited

Loading

eli-schwartz commented May 22, 2018 •

edited

Loading

bernd-wechner commented Jul 11, 2019 •

edited

Loading

bernd-wechner commented Jul 11, 2019 •

edited

Loading

martin3000 commented Jul 28, 2019 •

edited

Loading

verhovsky commented Jan 2, 2020 •

edited

Loading