Poetry is incompatible with unstable-tagged Python builds (Invalid PEP 440 version: '3.Y.Z+')

[dimaqq]

I'v just installed Python3.11 from deadsnakes/nightly.

Poetry can't use it, because the Python version has a + in it.

Invalid PEP 440 version: '3.11.0+' is all I get 😭

P.S. if the wise men declare that deadsnakes is at fault here, I'll happily bug dead snakes.

[neersighted]

Hmm, it looks like this could be a parse error with the Python version numbers provided by Deadsnakes? Looking closer at #6334, it looks like this might be a dupe. That issue was closed as a environment issue caused by bad versions in the target environment, but looking closer, that only represents the comments. The issue body, like this one, seems to be about a Python version.

Deadsnakes versions tend to be like 3.11.0+jammy1, which should be valid; some investigation of the actual version numbers from sys would be quite helpful. We might have to add some logic to strip extra tokens from Python versions in any case -- they really don't make sense for interpreter version comparisons as we don't support anything other than simple three-part-tuple versions anyway.

[dimaqq]

>>> import sys
>>> sys.version
'3.11.0+ (main, Oct 30 2022, 08:56:33) [GCC 9.4.0]'
>>> sys.version_info
sys.version_info(major=3, minor=11, micro=0, releaselevel='final', serial=0)

[neersighted]

PTAL @radoering @dimbleby

[radoering]

In the stack trace you can see that python_full_version is the culprit. According to PEP 508 python_full_version is equal to platform.python_version(), which is 3.11+ in this case. Although I could not find a direct reference, the sample values for python_full_version in PEP 508 indicate that it should be PEP 440 compliant.

Stack trace

Finding the necessary packages for the current system

Stack trace:

16  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/application.py:329 in run
      exit_code = self._run(io)

15  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/console/application.py:185 in _run
      exit_code: int = super()._run(io)

14  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/application.py:423 in _run
      exit_code = self._run_command(command, io)

13  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/application.py:465 in _run_command
      raise error

12  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/application.py:449 in _run_command
      exit_code = command.run(io)

11  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
      status_code = self.execute(io)

10  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/commands/command.py:83 in execute
      return self.handle()

 9  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/console/commands/install.py:146 in handle
      return_code = self.installer.run()

 8  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/installation/installer.py:115 in run
      return self._do_install()

 7  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/installation/installer.py:320 in _do_install
      with solver.use_environment(self._env):

 6  /usr/lib/python3.8/contextlib.py:113 in __enter__
      return next(self.gen)

 5  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/puzzle/solver.py:65 in use_environment
      with self.provider.use_environment(env):

 4  /usr/lib/python3.8/contextlib.py:113 in __enter__
      return next(self.gen)

 3  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/puzzle/provider.py:174 in use_environment
      self._python_constraint = Version.parse(env.marker_env["python_full_version"])

 2  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/core/version/pep440/version.py:182 in parse
      return parse_pep440(value, cls)

 1  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/core/version/pep440/parser.py:83 in parse_pep440
      return PEP440Parser.parse(value, version_class)

InvalidVersion

Invalid PEP 440 version: '3.11.0+'

at ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/core/version/pep440/parser.py:69 in parse
     65│     @classmethod
     66│     def parse(cls, value: str, version_class: type[T]) -> T:
     67│         match = cls._regex.search(value) if value else None
     68│         if not match:
  →  69│             raise InvalidVersion(f"Invalid PEP 440 version: '{value}'")
     70│
     71│         return version_class(
     72│             epoch=int(match.group("epoch")) if match.group("epoch") else 0,
     73│             release=cls._get_release(match),

[radoering]

I re-read PEP 508 and there seems to be an escape hatch for non-compliant versions:

The <version_cmp> operators use the PEP 440 version comparison rules when those are defined (that is when both sides have a valid version specifier). If there is no defined PEP 440 behaviour and the operator exists in Python, then the operator falls back to the Python behaviour.

However, that implies python_full_version >= "3.9" is fulfilled for Python 3.11, but not for deadsnakes Python 3.11+. (IIUC, a version comparison should be used for the first one and a string comparison for the second one.) That can't be the intention of deadsnakes authors!? Maybe, I miss something but it would probably be better to use PEP 440 compliant versions. Even if poetry would support these weird versions, the consequences would be unexpected for most users. 🤔

[dimbleby]

Appending a '+' seems to be the convention in the python source

https://github.com/python/cpython/blob/a9a8c8712665377cfa83af4b632b0db529ec1853/Include/patchlevel.h#L26 is presumably the origin of this

you could try persuading them that using a PEP440-compatible version would be more sensible

[dimbleby]

python/cpython#88166

looks like that became #4021, but the reporter closed it just twenty minutes after opening it...

[bellini666]

I have the same issue using python 3.11 packaged inside the official debian repositories. My python3.11 --version output is Python 3.11.0+

[dimaqq]

Btw., deadsnakes Python 3.10 also suffers this fate.

My current workaround: sudo hexedit /usr/bin/python3.10 find the version and overwrite the + with a zero byte.

And same for 3.11.

🙏🏻 project maintainers, please consider this important 🙏🏻

[neersighted]

I think it's obvious that deadsnakes in general is broken with Poetry as both 3.11 and 3.10 are GA versions.

[dimbleby]

more like: choosing to use the nightly builds for python 3.10 and python 3.11 is the wrong choice for almost everyone at this point, since there are stable releases available.

[neersighted]

@dimaqq, I think you misunderstood my meaning -- if 3.11 is broken, it is reasonably obvious that 3.10 is also broken, as they are both GA versions (so the + is not a transitive property of a pre-release build).

[dimbleby]

the regular 3.10 build is not 'broken' and I imagine neither is 3.11, rather folk are choosing to use nightly builds.

I get my 3.10 from deadsnakes myself, and the versioning is completely normal:

$ cat /etc/apt/sources.list.d/deadsnakes-ubuntu-ppa-focal.list
deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal main
# deb-src http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal main

$ python3.10 --version
Python 3.10.8

$ python3.10 -c 'import platform ; print(platform.python_version())'
3.10.8

that still leaves poetry confused by version numbers from nightly builds: but nearly no-one should have any reason to use those, certainly for pythons 3.10 and 3.11

[neersighted]

Ah, that would have been nice to have included in the reproduction and adds important context; the severity of this is much lower than I believed based on that.

[dimbleby]

to be fair it is the very first sentence of the very opening of this thread

[bellini666]

The official package from the current Debian Testing and Unstable (https://tracker.debian.org/pkg/python3.11) has this issue, and it is not using nightly builds.

❯ which python3.11
/usr/bin/python3.11

❯ dpkg -S /usr/bin/python3.11
python3.11-minimal: /usr/bin/python3.11

❯ python3.11 --version
Python 3.11.0+

[dimbleby]

um, it obviously is using nightly builds. You can tell by the version number!

probably "testing and unstable" is the clue

[neersighted]

Indeed, I am not worried about that case as much:

• python3.11 is explicitly experimental in unstable/testing; Debian does not ship Python versions in parallel under normal circumstances
• Unstable/testing are moving targets and not reasonable environments for 95% of users; those who are using them are theoretically equipped to understand and mitigate this kind of transient issue
• Debian Python packaging is always idiosyncratic and bordering on broken; there is a maintainer who is invested in papering over decisions made by Debian, but he has had little time lately and no one else is invested in it

[bellini666]

um, it obviously is using nightly builds. You can tell by the version number!

Just checked and you are right. It is not using the actual final but some other commit from the 3.11 branch. Sorry about assuming it was, the packages from debian usually use official released tarballs.

probably "testing and unstable" is the clue

Not really. The name is misleading, but testing is just an alias to the codename that will become the next stable version, and unstable is where packages get uploaded and automatically get migrated to the testing one if not issues are found. The debian team doesn't usually upload nightly versions to any of those repositories (there's a experimental repository for that there), and those packages will be the ones in the final stable release (just as they are, no repacking or recompiling) as long as no critical issues are found and no new versions are released and packaged.

[stefanor]

Debian Python person here:

python3.11 is explicitly experimental in unstable/testing; Debian does not ship Python versions in parallel under normal circumstances

It's in the process of becoming supported. Hopefully for the next stable release, but that depends on how much of the world works with it, and whether we can fix the rest in time.

Unstable/testing are moving targets and not reasonable environments for 95% of users; those who are using them are theoretically equipped to understand and mitigate this kind of transient issue
Debian Python packaging is always idiosyncratic and bordering on broken; there is a maintainer who is invested in papering over decisions made by Debian, but he has had little time lately and no one else is invested in it

I wouldn't really agree with either of these sentiments. But bugs aren't the place to litigate that. I really don't think Debian did anything wrong here. Poetry didn't expect to see a + in a Python version, but they are now something to be expected, since 3.11. That's about the extent of the issue, here.

[neersighted]

It's in the process of becoming supported. Hopefully for the next stable release, but that depends on how much of the world works with it, and whether we can fix the rest in time.

Right, but 3.10 is currently the primary Python version, and will be removed if there is a cut-over to 3.11, right?

but they are now something to be expected, since 3.11. That's about the extent of the issue, here.

Is that a statement of intent by Debian? The + signifies it's not a final build as tagged by upstream -- does Debian intend to ship a 3.11 based on an untagged commit?

[stefanor]

Right, but 3.10 is currently the primary Python version, and will be removed if there is a cut-over to 3.11, right?

Yes, it'll be removed when possible.

Is that a statement of intent by Debian? The + signifies it's not a final build as tagged by upstream -- does Debian intend to ship a 3.11 based on an untagged commit?

We regularly ship a patch bringing in all the latest commits in the maintenance branch, not just the tagged releases.
That's not my decision, so I don't know the full thinking there, but we often need those patches, because they resolve issues we're facing.

[stefanor]

but they are now something to be expected, since 3.11

To be clear, I was saying this is now something to be seen, in general. Not specifically in Debian. It's common to be working with Python built out of the source tree (when developing cpython itself, or chasing down bugs in it, for example).

I don't think Poetry should restrict itself to functioning correctly on release tags of cPython.

[neersighted]

I don't think Poetry should restrict itself to functioning correctly on release tags of cPython.

I don't think anyone disputes that, but this is certainly not as hot a fire as we originally thought. PRs are welcome but this will likely be moderately invasive/painful to fix.

[eamanu]

Hi, what about something like this? (it works in my local tests in Debian)

Index: src/poetry/utils/env.py
===================================================================
--- src/poetry/utils/env.py	2022-11-30 20:30:11.425524136 -0300
+++ src/poetry/utils/env.py	2022-11-30 20:30:31.829803450 -0300
@@ -1618,7 +1618,7 @@
             "platform_release": platform.release(),
             "platform_system": platform.system(),
             "platform_version": platform.version(),
-            "python_full_version": platform.python_version(),
+            "python_full_version": platform.python_version().rstrip("+"),
             "platform_python_implementation": platform.python_implementation(),
             "python_version": ".".join(platform.python_version().split(".")[:2]),
             "sys_platform": sys.platform,

[neersighted]

That's far from the only place where we request the full Python version, and we'll still need to create robust tests. It's also very painfully hardcoded and without explanation outside blame; ideally we solve this in the code that actually parses the version.

[radoering]

I wouldn't strip the "+" in general in the code parsing versions because it's definitively not valid for package versions (and at least a bad choice for the python version). I haven't searched for other places yet but remembering the failure on 3.11+ I think that is at least part of a correct fix. Of course we should add a comment why we are stripping the "+" and a test would be nice.

[neersighted]

My thinking is we should centralize it using a parse_python_version() variant that encapsulates the stripping of the + so we don't randomly proliferate it in the codebase and get burned by a future omission. The parse_python_version() code can just be a preprocessing step before we hand it to our regular version parsing code, and can be unit tested; we should also end-to-end test the main usage (in env.py).

[brechtm]

Not sure whether it brings anything useful to the table, but the Python 3.12.0a2 'release' (installed using pyenv) also reports its version as 3.12.0a2+ and thus fails to work with Poetry.

[brechtm]

BTW, a simple workaround is to first create the virtualenv using the venv package (from the Python version you want to use), which Poetry will happily use.

[jshwi]

This happens for me if run poetry install without an existing environment if i installed poetry using curl -sSL https://install.python-poetry.org | python3 - --force while under pyenv global 3.x-dev

If I use poetry env use 3.y this works, but as long as 3.y-dev is not installed under pyenv

I have to be in a specific version, patch included, e.g. 3.8.13 when running curl -sSL https://install.python-poetry.org | python3 - --force for my for poetry to work by default, without needing to use poetry env use 3.<some_other_minor_version> as long as 3.<some_other_minor_version>-dev is not installed

The workaround for me is

• pyenv install 3.8.13
• pyenv uninstall 3.8-dev
• pyenv global 3.8.13
• curl -sSL https://install.python-poetry.org | python3 - --force

I am on macos

[yssource]

the same problem happens to me.

pyenv install 3.10-dev

Invalid PEP 440 version: '3.10.8+'

[magniff]

The same thing on 3.11-dev

$ poetry run python --version
Python 3.11.2+

[neersighted]

There is no need to comment with a 'me too' here -- this will be true for any non-release build of Python, so 3.x-dev will be universally affected.

[davnat]

There is no need to comment with a 'me too' here -- this will be true for any non-release build of Python, so 3.x-dev will be universally affected.

Maybe a 'me too' comment is not that useful from a development point of view, but it' is a way to express that this issue is affecting many users.

I understand that fixing this will be invasive/painful, but I can't agree that this is so low priority as it seems to be for you: Poetry is a development tool and IMHO it should be expected that users of Poetry use whatsoever Python version.

A quick note on Debian, too.
If you want/need an up to date system, 90% of the time you can't use Debian stable: you want testing, and you'll probably install selected packages from experimental and/or PPAs. This is physiologically due to the "release when ready" Debian policy.
I think the vast majority of Debian users are running testing most of the time (except when just released, obviously).

[neersighted]

"Me too" is wholly unproductive -- GitHub has 👍 for that, which both is surfaced in the UI as a voting method, and avoids sending unnecessary notifications. Browbeating the volunteer maintainers of a project is not the way to get an issue prioritized; typically the way to do that is by sending a patch.

#7462 is under review as one possible approach; if you want to see things move faster I would suggest sending your own PR with a combination of the feedback from #6925 (comment) and #7462 (comment) incorporated.

I'm sorry that you feel Poetry is defective for your needs because of this issue; Poetry is not unique here, and as you can see from the linked issues, this is something of an ecosystem-wide problem. Regrettably, there is not infinite time in the day, and no one is paid to work on Poetry or sells you any sort of support. The best we can do is collaborate to improve the tool, and make things as actionable as possible so that maintainer time can be used as efficiently as possible.

[Amar1729]

Sorry for the noise to poetry maintainers, but as a general notice for pyenv users finding this issue ( @magniff @yssource @jshwi ): a pyenv python version X.Y-dev means development, and will have the + suffix. However, as the readme notes, most commands will auto-resolve to the latest version.

So if you run pyenv install 3.8, pyenv will install the latest 3.8.z (no +), and if you run pyenv global 3.8, pyenv will set the latest installed 3.8.z as your global. As *-dev will always be the latest if you have it installed, just make sure not to install a -dev release, and you should be good.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.