Ubuntu 22.04, Python 3.10, and broken virtual environment layouts

[colindean]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu Jammy

• Poetry version: 1.2.0

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/colindean/27d36dcc7cf78f722f374ebd5bfd970f (also contains log output and a brief description of env setup steps)

Issue

root@80d0e404badc:/what# poetry self add poetry-plugin-bundle
Using version ^1.0.0 for poetry-plugin-bundle

Updating dependencies
Resolving dependencies... (3.8s)

[Errno 2] No such file or directory: 'python'

I believe the problem is in here:

poetry/src/poetry/utils/env.py

Lines 1231 to 1248 in f1af3f8

	def _find_python_executable(self) -> None:
	bin_dir = self._bin_dir
	if self._is_windows and self._is_conda:
	bin_dir = self._path
	python_executables = sorted(
	p.name
	for p in bin_dir.glob("python*")
	if re.match(r"python(?:\d+(?:\.\d+)?)?(?:\.exe)?$", p.name)
	)
	if python_executables:
	executable = python_executables[0]
	if executable.endswith(".exe"):
	executable = executable[:-4]
	self._executable = executable

Ubuntu doesn't provide a python command anymore, just python3:

root@80d0e404badc:/what# python --version
bash: python: command not found
root@80d0e404badc:/what# python3 --version
Python 3.10.4

Seeing as though all supported versions of Python on which Poetry runs should also provide a python3 executable, perhaps the fallback here should be python3?

poetry/src/poetry/utils/env.py

Line 1178 in f1af3f8

self._executable = "python"

[colindean]

N.b. I hadn't even run poetry install yet and just now went I did…

root@80d0e404badc:/what# poetry install
Creating virtualenv what--AkdOZe_-py3.10 in /root/.cache/pypoetry/virtualenvs

[Errno 2] No such file or directory: 'python'

[colindean]

I appear to be unable to tell poetry which Python to use, either:

root@80d0e404badc:/what# poetry env use $(which python3)

[Errno 2] No such file or directory: 'python'

[neersighted]

I think this is actually the old DEB_PYTHON_INSTALL_LAYOUT=deb issue -- can you set that in the environment and reinstall Poetry/poetry install from scratch? python will always exist as far as Poetry is concerned as it is called for by the virtual environment spec.

[colindean]

root@80d0e404badc:/what# export DEB_PYTHON_INSTALL_LAYOUT=deb
root@80d0e404badc:/what# poetry self add poetry-plugin-bundle
Using version ^1.0.0 for poetry-plugin-bundle

Updating dependencies
Resolving dependencies... (16.5s)

[Errno 2] No such file or directory: 'python'

[neersighted]

We're still trying to fully understand how we can work around the regression in Debian/Ubuntu as they don't appear willing to walk back the changes. I am not sure why you're having that issue with self add, but I can confirm for you that on the ubuntu:22.04 container with the python3 package I can install Poetry, run the Poetry test suite (after poetry install), and poetry self add without issue if I set DEB_PYTHON_INSTALL_LAYOUT=deb before running any Python.

It is important to note that the issue is not in Poetry, but is in the virutualenv package which we consume. It is still considered unsolved there, as the changes made by Debian/Ubuntu result in an invalid virtual environment according to the spec: pypa/virtualenv#2350

[colindean]

OK. To confirm, does DEB_PYTHON_INSTALL_LAYOUT=deb need to stay forever, then? As in, rather that just needing for it to be set at installation time, it should be set the user's shell rc or profile?

[neersighted]

From a practical standpoint, it's the known workaround right now. Fundamentally this is unsolved and upstream seems intransigent or indifferent (see: https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920).

The deadsnakes PPA solves this with a patch on top of the Debian 3.10 patches. Ideally this would get solved in virtualenv, but things are stalled there too. We haven't put too much effort into figuring out how to work around it in Poetry (but the solution definitely is NOT just adapting to local/bin as that is a violation of the virtual env spec and other tools will break), if that is even possible without somehow trying to monkey-patch the virtualenv module.

The best solution is to use a Python.org Python (aka a build from source installed into /opt/ or using python-build, or pyenv or a python docker hub image, or the like), or to use a deadsnakes build, until some upstream solves this, or we figure out a not-awful way to work around it here in the downstream.

[colindean]

Thanks for the expansion. I understand now. I encountered this problem while trying Poetry 1.2.0 in a container to test how installation of plugins would work (globally at Poetry level or per-project), as I eventually went to Discord to discuss!

Thanks for your time on this, I really appreciate it.

[colindean]

I should add that my team's normal way of managing our Pythons is through pyenv for our workstations and conda for our production containers. So, this issue isn't a major blocker for us!

[garar]

Hi!

Is there a workaround to this issue? I've tried using DEB_PYTHON_INSTALL_LAYOUT=deb variable when using poetry add but it seem that it doesn't work. Is there some other way of using DEB_PYTHON_INSTALL_LAYOUT=deb?

Thanks!

[neersighted]

You should ideally set it before ever installing Poetry -- however, I am reasonably sure that setting it during env creation (e.g. poetry env use, poetry add without an existing env, etc) and during subsequent poetry install/poetry add/poetry update) should be sufficient. I do not work regularly with a Ubuntu/Debian distro, so I am guessing to an extent (I just export it in .bashrc when it comes up).

[AdamWRichardson]

I also ran into this problem and didn't know about the DEB_PYTHON_INSTALL_LAYOUT=deb workaround so will try that now. On my digging I may have found some more useful information as I too thought it was a problem with the way poetry handles the system python but the trace suggests otherwise.

The first "poetry error" (by which I don't mean that poetry is doing anything wrong) in the trace comes from the env_manager.create_venv() command which, as the name suggests, creates a new venv using python3-venv. When I have a look at this new venv it contains <venv_path>/share and <venv_path>/lib but no <venv_path>/bin which should contain the python executables as well as the python symlink.

This seems very weird because when I create the venv myself the bin directory does exist so I'm not sure what's happening internally for this bin directory to not get made.

Like I say I hope this is helpful to others and helps any debugging/patching effort either here or in Ubuntu upstream. Will try the environment variable setup now

[yrro]

I found that installing Debian's python3-poetry results in a perfectly working Poetry. Looking at https://sources.debian.org/src/poetry/1.3.2%2Bdfsg-3/debian/patches/ I don't believe this is because they've patched Poetry themselves.

[gaby]

I have been trying for weeks to get Poetry working with Ubuntu 22.04:

Tried:

• pip3 install poetry
• apt install python3-poetry
• curl -sSL https://install.python-poetry.org | python3 -

Running anything related to poetry simply fails, this is a brand new Ubuntu 22.04 VM

[nogjam]

Had this issue as well inside VSCode's integrated terminal. It seems VSCode does some kind of venv activation automatically when starting up the integrated terminal, which you can get out of with the deactivate command. However, it seems this is not Poetry's "preferred" method of venv usage, which is through either poetry run ... or poetry shell. In the latter case, you can get out with exit. Not sure what VSCode is doing there, but probably good to be aware of if you're using it.

The end result in my case was that removing Poetry (1.6.1) and then reinstalling Poetry (1.7.1) with the install script solved my issue. After I had done so, however, I noticed I was still in the venv in VSCode's integrated terminal. I wonder what would have happened if I had used deactivate and then tried poetry env use again, which is where I was seeing this bug manifest.

[jorritsandbrink]

Tried many things suggested here and in other places, but nothing worked. Eventually removing the cache as suggested in this comment did the trick:

rm -r ~/.cache/pypoetry