PEP 440 parse exception on packages in /usr/lib/python3/dist-packages/ installed by ubuntu

[pae23]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: ubuntu 22.04
• Poetry version: latest
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/pae23/631564ff9bbe1efd0be33ef107f67b90

Issue

At least some packages installed by Ubuntu in /usr/lib/python3/dist-packages/ are not PEP440

For example :

# apt-cache policy devscripts
devscripts:
  Installed: 2.22.1ubuntu1
  Candidate: 2.22.1ubuntu1
  Version table:
 *** 2.22.1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

I do this :

apt install devscripts
poetry config virtualenvs.options.system-site-packages true
poetry install -vvv

And I get this error :

  InvalidVersion

  Invalid PEP 440 version: '2.22.1ubuntu1'

  at /usr/local/lib/python3.10/dist-packages/poetry/core/version/pep440/parser.py:69 in parse
       65│     @classmethod
       66│     def parse(cls, value: str, version_class: type[T]) -> T:
       67│         match = cls._regex.search(value) if value else None
       68│         if not match:
    →  69│             raise InvalidVersion(f"Invalid PEP 440 version: '{value}'")
       70│ 
       71│         return version_class(
       72│             epoch=int(match.group("epoch")) if match.group("epoch") else 0,
       73│             release=cls._get_release(match),

I know Ubuntu doesn't respect PEP440 but I think we need a workaround to disable PEP440 check on packages installed in /usr/lib/python3/dist-packages/

[finswimmer]

Hello @pae23,

I'm afraid there's nothing Poetry can do about it. For proper dependency resolving it is necessary to "understand" the version number of a Package. If a package doesn't follow PEP-440 this is not possible.

Please see also my comment in #2167 (comment)

My advice in your case would be to not use the system-site-packages.

fin swimmer

[jonathan-boudreau-work]

How was poetry able to parse it before then? It works fine in version 1.1.15. The regression seems to be only present in 1.2.x.

[neersighted]

How was poetry able to parse it before then? It works fine in version 1.1.15. The regression seems to be only present in 1.2.x.

Poetry 1.2 has a much more robust (and generally faster!) installer compared to 1.1. When Poetry installs your project, it needs to reason about what is already installed in the environment. Instead of merely doing the equivalent of installing a requirements.txt, Poetry actually introspects the state of the target environment (into a datastructure known internally as the InstalledRepository) and comes up with a plan to mutate the state of the environment to match the lock file.

Because Poetry needs to do this introspection of the environment (and it reuses the Repository code, which is battle-tested and one of the primary datastructures/abstractions in Poetry), it must parse and understand version numbers. This goes through the regular PEP 440 code path, which is the standard for the Python ecosystem.

Normally non-PEP 440 version numbers are very hard to achieve as all high-quality Python tooling (including PyPI) will block the use of non-PEP 440 version numbers. However, distro packagers may alter version numbers in ways not compatible with standard Python tooling. These packages are installed and detected by Poetry as they are provided by the distro package manager and installed using non-Python tooling.

Ultimately, there is nothing Poetry itself can do about the presence of these packages, as ignoring them is dangerous and could lead to a mismatch with the lockfile environment, and adding some bailout for distro versions is also dangerous as Poetry could overwrite operating system files.

The correct move is to not install an application into the system environment, but instead install it into a virtual environment -- thus isolating it from any version numbers that are incompatible and provided by the host OS. Even in a container, a virtual environment provides an important layer of separation, as Python imports are versionless and there is (very little) support for an import hierarchy -- things are flat, more or less.

[danieldjewell]

Although this is closed and although the position appears to be one of "not our problem" (indicated both here and in #6334 by @dimbleby) - I would humbly offer a minor comment/observation:

Yes, it's important for everyone else to follow standards - that's part of the purpose of standards... and they only work if people follow them! That said, given that, within the Python ecosystem, there is not a regulatory governmental body with the power to enforce standards (e.g. the FCC/FAA in the US, Ofcom in the UK, BNA/Bundesnetzagentur in DE, etc.) ... one must take a slightly more pragmatic approach, in my very humble opinion.

While I completely understand and agree that it would indeed be an impossible task to make changes to accomodate every possible exception, I do not believe it wise (as a general principle of software design) to categorically and unconditionally dismiss anything as someone else's problem or fault.

That said, as @neersighted points out, we know that distributions (and/or other developers) are making changes to the version numbering making them incompatible with PEP 440. While they ought not to do this, they are. That's the simple reality. Am I (or even a group of developers) going to ever have a chance of "fixing" this? No. And I don't think I'm being defeatist - I believe it's a realistic assessment of what is most likely to happen (or not happen). I admire Poetry's commitment to and compliance with standards - it's a great thing. But again, it only works if everyone follows them.

As a note to the above - The Official Debian Policy Manual describing Control files and their fields - 5.6.12 Version specifically allows verison numbers to contain characters prohibited by PEP 440 (e.g. tilde [~], plus [+], hyphen [-], etc.) ... further, the Debian Python Packaging Manual makes no mention of PEP 440. I actually can't find any existing policy guidance from Debian on PEP 440. While it does seem like the policy ought to be updated, that's not how it stands right now.

As a result, I think that it's important to make decisions not just on the basis of how things ought to be, but rather to take into consideration how things actually are. Perhaps there is a relatively simple way for Poetry to be slightly more flexible - not to address every single exception to PEP 440 but perhaps a few lines of code could address the majority of the exceptions? Perhaps failing more "gracefully" with a prompt/warning to the user that Poetry is about to make an assumption (in violation of PEP 440) could ease the pain that users feel?

[neersighted]

I don't think that the stance that we will have to require PEP 440 versions is unreasonable from either a technical or social perspective:

• Poetry is designed to reproduce a Python environment that your project requires deterministically across platforms. This requires Poetry to be able to reproduce the same results across platforms and to ensure the environment it installs into matches the saved result.
• Poetry is designed to interface with the 'mainstream' Python packaging ecosystem, which does enforce, both socially and technologically, PEP 440.
• The PyPA is a standards-setting body that makes decisions about the recommendations and enforced standards for the 'mainstream' Python packaging ecosystem -- to be compatible with other tooling Poetry must focus on PEP 440 compatibility.
• If Poetry were to ignore unparsable versions, it would effectively be blind to packages in your environment, including packages that conflict or are out of date according to the solution Poetry creates.
• If Poetry were to attempt to special-case and 'parse' Debian versions, it would still not answer whether those versions are considered compatible (or whether to update them, and break your package manager's assumptions about provenance).

In short, to ignore or try to parse versions that do not follow a set syntax (with known rules for order and compatibility) is fraught. The reason Poetry requires PEP 440 versions is not to punish those who participate in other package ecosystems or enforce our own opinions -- it is because Poetry needs to transform one environment (the InstalledRepository) to match another (the LockedRepository) and needs to be able to reason about the state of the target environment in order to do so.

Debian and other alternate (e.g. non-PyPI/index) tooling are still free to do what they want, and Poetry is fully usable on these systems when using the isolation mechanisms provided by the Python ecosystem (i.e. virtual environments) to prevent mixing of two different 'worlds.'

[mstandley-tempus]

Here's the bug report opened against the wrongly versioned OS package: https://bugs.launchpad.net/ubuntu/+source/python-debian/+bug/1926870

[jonathan-boudreau-work]

Maybe if there are potential conflicts a warning message could be given. Another idea would be to have an option (config or cli flag) to ignore potential conflicts when it isn't parseable. Crashing isn't the only option here.

Its very concerning that poetry seems to have no issue making breaking changes without any mention in the changelog. Currently we are pinned to the old version that still works, and are looking for alternatives.

[neersighted]

To parse non-compliant versions would require very invasive and had to test changes in poetry-core. It's certainly possible, but it seems entirely fraught when the fundamental issue is tooling external to Poetry and the regular packaging ecosystem injecting versions unexpectedly. While a carve-out could solve the immediate issue, I fear it makes support much harder in the long run.

To advocate for a change in Poetry's parsing behavior, I would suggest presenting a use case that cannot be solved by virtual environments and that requires colliding with the system-provided dist-packages.

As far as breaking changes go, it is captured in the changelog of poetry-core, but we maintain that separately and do not roll it into Poetry's, especially as historically the poetry-core dependency has been semver and not pinned. We're still trying to work out the best way to manage the release cycle of these two separate-but-closely-related components.

[muppetjones]

To advocate for a change in Poetry's parsing behavior, I would suggest presenting a use case that cannot be solved by virtual environments and that requires colliding with the system-provided dist-packages.

I think you don't need to look very far. Simply look ask "how do I ..." for nearly any problem in python, and you'll see that there are many options within python itself. Look no further than __version__. Edit: I know this isn't a direct use case. Simply pointing out that there is so much variation in how python is used that forcing a specific path is not really a viable option.

Bottom line: Conventions and compliance are great, but when there's a long history without, adoption must be flexible. From where I stand, this looks like a tipping point for poetry. The issues surrounding 1.2 version have many of my coworkers starting to suggest that we stop using poetry. I don't think that the pros yet outweigh the cons, but we are actively rolling back from 1.2.

[seano-vs]

I've found this thread after facing an enormous amount of PEP 440 issues myself (I have ubuntu packages installed).

I'm a long time poetry fan, but I'm surprised this decision was made with regards to alienating the grand majority of Ubuntu users. This was a really hard stance to take without providing any way around it. I appreciate how you are trying to push the industry into using a better standard, but "don't use that package" or "Tell Canonical to abandon their standard" isn't a viable solution for, well, anyone.

[jedie]

YunoHost version scheme looks like: 0.16.0~ynh1 (Where 0.16.0 is the upstream version and ~ynh1 a suffix of the YunoHost package) ... Because of this hard PEP 440 error, i can't use Poetry v1.2.x now.

I would have expected that first a WARNING is emitted, and non PEP 404 versions are still allowed for a while. But with 1.2.0 this breaks things.

[gingershaped]

This problem prevents me from using Poetry reliably on my Raspbian Linux system. If I turn off system-site-packages I can't use things like gi which are next to impossible to install in a virtualenv, but if I leave it on this error occurs.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.