RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing #215

nschloe · 2021-01-28T16:15:16Z

When running

python3 -m build --sdist --wheel .

on Mac OS, I'm getting

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 210, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 202, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 81, in build_package
    _build_in_isolated_env(builder, outdir, distributions)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 45, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/env.py", line 55, in __enter__
    executable, pip_executable = _create_isolated_env(self._path)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/env.py", line 171, in _create_isolated_env
    executable = _find_executable(path)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/env.py", line 208, in _find_executable
    raise RuntimeError('Virtual environment creation failed, executable {} missing'.format(executable))
RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing

Not sure at all whe it would be looking for python in /usr/local. Any idea what's going wrong?

The text was updated successfully, but these errors were encountered:

gaborbernat · 2021-01-28T16:27:29Z

What folder are you running this in?

nschloe · 2021-01-28T16:32:34Z

A subfolder of $HOME, where the software sits (complete with pyproject.toml etc.).

layday · 2021-01-28T17:36:59Z

How did you install Python? It looks like sysconfig might've been patched to ignore the value of base.

nschloe · 2021-01-29T08:00:33Z

Using default Python on MacOS 11.1 here. Hm...

uranusjr · 2021-01-29T08:45:25Z

default Python on MacOS 11.1

macOS 11.1 does not have a default Python. (Well, technically there is, but it’s not usable in user land, and certainly not located at /usr/local/bin/python.)

nschloe · 2021-01-29T08:47:17Z

Then I have no idea. It resides in

which python3

/usr/bin/python3

layday · 2021-01-29T13:14:39Z

I wonder if sysconfig from another Python installation is on your sys.path. Not sure if it'll get us very far, but could you attach the output of these two commands:

python -c "import sysconfig; print(sysconfig.get_paths())"
python -c "import sys; print(sys.path)"

nschloe · 2021-01-31T11:48:32Z

I played around with it a little more and I'd say this is too hard to figure out remotely. Thanks anyways for your help! I'll post a solution when I found one.

bryanheinz · 2021-02-06T16:37:17Z

@nschloe were you doing this on an M1 Mac? i ran into this issue last night on an M1 Mac, but ran fine when i tried on an Intel Mac.

nschloe · 2021-02-07T08:50:43Z

It's an old Intel MacBook Air, but fully upgraded (Big Sur 11.1).

Patol75 · 2022-01-19T10:21:10Z

I am encountering a similar issue on Ubuntu 20.04.3 LTS using Python 3.10, which I installed from the deadsnakes PPA.
python3.10 -m build returns

* Creating venv isolated environment...

Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 372, in main
    built = build_call(
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 229, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
  File "/home/thomas/.local/lib/python3.10/site-packages/build/env.py", line 104, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/env.py", line 258, in _create_isolated_env_venv
    executable, script_dir, purelib = _find_executable_and_scripts(path)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/env.py", line 303, in _find_executable_and_scripts
    raise RuntimeError(f'Virtual environment creation failed, executable {executable} missing')
RuntimeError: Virtual environment creation failed, executable /tmp/build-env-lcuno53i/local/bin/python missing

ERROR Virtual environment creation failed, executable /tmp/build-env-lcuno53i/local/bin/python missing

If, instead, I use Python 3.9, also installed from the deadsnakes PPA, python3.9 -m build completes without errors.

layday · 2022-01-19T10:41:33Z

It looks like the prefix scheme is changed to posix_local as a result of this patch and build expects that venvs are created using the default (prefix) scheme. However, virtualenv has added support for the venv scheme and the same patch includes:

+_INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_prefix']

so venvs are created using a different scheme.

layday · 2022-01-19T10:46:02Z

We could add another condition here to check for the existence of the venv scheme although I'm -1 on supporting venv before it lands in CPython.

layday · 2022-01-19T10:49:05Z

~~@gaborbernat Can we not grab the Python executable path from virtualenv directly?~~

Wait, this is the stdlib venv... so something else is going on.

Oh right, of course, the venv module doesn't use sysconfig, so deadsnakes would need to patch venv if they are using an incompatible default scheme.

Please report this to deadsnakes.

Patol75 · 2022-01-19T22:25:49Z

@layday As is tracked above, I have reported the issue to deadsnakes, but the developer there believes it is actually related to build. I can confirm that I can replicate the behaviour he documents in his reply, as well as actually activating the virtual environment and deactivating it.

layday · 2022-01-19T22:49:40Z

Yes, venvs are created "correctly". The issue is that we cannot extract the venv paths if a different scheme is used for the venv because we operate on the assumption that the system default and the venv schemes are the same. That is how venvs have historically worked and it's how they are supposed to work. If Debian or deadnakes or Fedora or whoever wish to deviate from the spec, they need to put in the effort to make sure their changes are compatible with the wider ecosystem.

layday · 2022-01-19T22:52:41Z

Can you actually pip install anything in this venv with the latest version of pip? If you can, i.e. if pip falls back on distutils, then it should also emit a warning.

Patol75 · 2022-01-19T23:13:19Z

Thanks for your help @layday.

Does the following change anything to your argument?

thomas@thomas-GL502VM:~$ python3.10 -m venv venv
thomas@thomas-GL502VM:~$ source venv/bin/activate
(venv) thomas@thomas-GL502VM:~$ python3 -m pip install --upgrade pip
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (21.2.4)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3.1
(venv) thomas@thomas-GL502VM:~$ python3 -m pip install --upgrade numpy
Collecting numpy
  Using cached numpy-1.22.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Installing collected packages: numpy
Successfully installed numpy-1.22.1
(venv) thomas@thomas-GL502VM:~$ ls venv/lib
lib/   lib64/ 
(venv) thomas@thomas-GL502VM:~$ ls venv/lib/python3.10/site-packages/
_distutils_hack  distutils-precedence.pth  numpy  numpy-1.22.1.dist-info  numpy.libs  pip  pip-21.3.1.dist-info  pkg_resources  setuptools  setuptools-58.1.0.dist-info
(venv) thomas@thomas-GL502VM:~$ deactivate
thomas@thomas-GL502VM:~$

layday · 2022-01-19T23:26:11Z

I assume pip calls get_preferred_scheme in 3.10; deadsnakes is varying the prefix depending on whether it's being called from inside a venv, meaning that it might or might not work for build depending on whether build itself was installed inside a venv. get_preferred_scheme wasn't introduced so people could fudge the paths at runtime.

Patol75 · 2022-01-19T23:45:10Z

Do you mind checking the latest reply from the deadsnakes developer? The thread is tracked above.

layday · 2022-01-19T23:50:03Z

We create venvs programmatically. We do not know where the venv Python is located, or where any of the other paths are located inside the venv, which is why we are calling sysconfig.get_paths in the first place. So no, whatever works for virtualenv and venv (or even pip) won't work for us. We could hardcode the Python path as venv-dir/Scripts/python on Windows and venv-dir/bin/python everywhere else, attempt to call get_paths in a subprocess and hope for the best. Personally, I'd rather Debian roll back their patches and work with the core Python team to find a workable solution.

gaborbernat · 2022-01-20T00:16:14Z

A workaround would be go install build[virtualenv], I think that would not have this issue 🤔

Patol75 · 2022-01-20T02:04:15Z

The developer at deadsnakes is wondering if an approach similar to this one could work for build.

layday · 2022-01-20T06:57:21Z

I mentioned that as a possibility earlier. I do think it's the wrong way to approach altering the venv scheme. If a different scheme is sought for venvs, a new entry should be added to the preferred scheme dict, for venvs. get_preferred_scheme is supposed to be the single source of truth for schemes by install type. And because it bears repeating: they should absolutely not be varying the schemes depending on the active prefix. I'm lukewarm to the idea of relying on an extension to the venv spec that has scarcely been discussed and has not been merged into CPython. But if it's gonna get us out of this bind, I suppose we could look for venv in the scheme names, like we do with the framework prefix for Apple's Python. Or we could recommend that people use virtualenv, like Bernat has suggested.

henryiii · 2022-01-20T14:11:31Z

Requiring build[virtualenv] is not an option. We can't force everyone to use the option to get a non-broken build. If we really can't get it properly fixed, we will be forced to mitigate it here.

@layday, can you try to engage the Debian developers (and/or any relevant parties) to see if this can be fixed? This is likely still fixable, since AFAIK it's only shipped to deadsnakes and probably Debian unstable.

FFY00 · 2022-01-20T19:53:52Z

The behavior is technically correct on Python 3.9 and older, it only becomes a problem when distributors break assumptions and change the install layout, which is not supported by the Python upstream. However, this becomes an issue in 3.10 as the Python upstream provides an official mechanism to customize the install layout.

Using the venv scheme would be possible, but that has not been standardized in the Python upstream, only Fedora currently implements it.

We can use the venv scheme if it exists, and fallback on the expected hardcoded path.

The deadsnakes developer can adopt the same approach of specifying the venv scheme if he wants. I do not want to interact with him, so this will be the full extent of my advice.

See https://bugs.python.org/issue45413 for more details.

hroncok · 2022-01-31T16:15:33Z

Using the venv scheme would be possible, but that has not been standardized in the Python upstream, only Fedora currently implements it.

That was proposed in #434

python/cpython#31034 is the CPython PR that adds it -- if merged, others can start using it soon (it should even fix macOS Python once they update to 3.11)

hroncok · 2022-01-31T16:21:28Z

deadsnakes now defines the "venv" installation scheme as well: deadsnakes/python3.10@9b93d73

layday · 2022-01-31T16:36:52Z

Yes, deadsnakes implementation is what prompted the discussion. Sent with [ProtonMail](https://protonmail.com/) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Monday, January 31st, 2022 at 18:21, Miro Hrončok ***@***.***> wrote: deadsnakes now defines the "venv" installation scheme as well: ***@***.***(deadsnakes/python3.10@9b93d73) — Reply to this email directly, [view it on GitHub](#215 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWEFC5SLDFDFMAMAE33UY2ZJHANCNFSM4WXKMP2A). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you were mentioned.Message ID: ***@***.***>

linsomniac · 2022-05-19T22:37:53Z

A workaround would be go install build[virtualenv], I think that would not have this issue 🤔

Thank you for this, I was following the Python packaging documentation ( https://packaging.python.org/en/latest/tutorials/packaging-projects/ ) and got stuck with a similar error message, this was the first google hit on a search for it, and installing build[virtualenv] solved my problem.

petersilva · 2022-12-02T20:09:13Z

same thing here... we are told python3 setup.py bdist_wheel is "obsolete" and to use pythom -m build...
but that crashes it a totally stock (ubuntu 22.04, os provided python everything):


fractal% python3 -m build
* Creating venv isolated environment...

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 372, in main
    built = build_call(
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 229, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
  File "/usr/lib/python3/dist-packages/build/env.py", line 104, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/usr/lib/python3/dist-packages/build/env.py", line 258, in _create_isolated_env_venv
    executable, script_dir, purelib = _find_executable_and_scripts(path)
  File "/usr/lib/python3/dist-packages/build/env.py", line 303, in _find_executable_and_scripts
    raise RuntimeError(f'Virtual environment creation failed, executable {executable} missing')
RuntimeError: Virtual environment creation failed, executable /tmp/build-env-tcdplhrn/local/bin/python missing

ERROR Virtual environment creation failed, executable /tmp/build-env-tcdplhrn/local/bin/python missing
fractal%

henryiii · 2022-12-02T20:53:19Z

Hmm, wonder if it's only making python3 instead of python? venv isn't supposed to do that, but maybe it's been hacked a bit?

FYI, build works fine on stock Ubuntu 22.04 in a docker container:

$ docker run --rm -it ubuntu:22.04
# apt update && apt install -y python3-venv python3-pip git
# git clone https://github.com/pypa/build
# cd build/
# python3 -m pip install build
# python3 -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (flit-core >= 3.4)
* Getting build dependencies for wheel...
* Building wheel...
Successfully built build-0.9.0-py3-none-any.whl

henryiii · 2022-12-02T20:57:56Z

Ahah, the python3-build package is busted. Not sure that's our fault. It's based on 0.7.0, which also shows this problem. So Debian / Ubuntu should update it. 0.8.0 and 0.9.0 both work.

henryiii · 2022-12-02T21:00:45Z

As a workaround you can install python3-virtualenv, which fixes this (and might be the best fix for Debian / Ubuntu).

zoot-inge · 2023-09-01T15:36:26Z

I found: https://bugs.launchpad.net/ubuntu/+source/python-build/+bug/1992108 with quick fix

As per pypa/build#215 (comment) and https://bugs.launchpad.net/ubuntu/+source/python-build/+bug/1992108

* Build also needs venv to be explicitly installed * Install build via pip as the Jammy package is broken As per pypa/build#215 (comment) and https://bugs.launchpad.net/ubuntu/+source/python-build/+bug/1992108 * Use dash-compatible = instead of bash-specific == Incidental trivial improvement in several other source files: fix spacing in strings broken over multiple lines.

nschloe closed this as completed Jan 31, 2021

layday reopened this Jan 19, 2022

layday closed this as completed Jan 19, 2022

Patol75 mentioned this issue Jan 19, 2022

Unable to create a virtual environment with Python 3.10 deadsnakes/issues#198

Closed

layday mentioned this issue Jan 31, 2022

Custom default sysconfig INSTALL_SCHEME leads to superfulous "Virtual environment creation failed" #433

Closed

hartzell mentioned this issue May 26, 2023

getting error while running poetry show python-poetry/poetry#7854

Open

zoranbosnjak mentioned this issue Dec 5, 2023

ast-tool-py install procedure for offline server zoranbosnjak/asterix-tool#2

Closed

jmarshall added a commit to populationgenomics/analysis-runner that referenced this issue Aug 1, 2024

Install build via pip as the Jammy package is broken

101928a

As per pypa/build#215 (comment) and https://bugs.launchpad.net/ubuntu/+source/python-build/+bug/1992108

jmarshall mentioned this issue Aug 1, 2024

More accumulated dataproc Dockerfile changes (and miscellanea) populationgenomics/analysis-runner#700

Merged

RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing #215

RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing #215

Comments

nschloe commented Jan 28, 2021

gaborbernat commented Jan 28, 2021

nschloe commented Jan 28, 2021 • edited Loading

layday commented Jan 28, 2021

nschloe commented Jan 29, 2021

uranusjr commented Jan 29, 2021

nschloe commented Jan 29, 2021

layday commented Jan 29, 2021

nschloe commented Jan 31, 2021

bryanheinz commented Feb 6, 2021

nschloe commented Feb 7, 2021

Patol75 commented Jan 19, 2022

layday commented Jan 19, 2022

layday commented Jan 19, 2022

layday commented Jan 19, 2022 • edited Loading

Patol75 commented Jan 19, 2022

layday commented Jan 19, 2022

layday commented Jan 19, 2022 • edited Loading

Patol75 commented Jan 19, 2022

layday commented Jan 19, 2022

Patol75 commented Jan 19, 2022

layday commented Jan 19, 2022

gaborbernat commented Jan 20, 2022

Patol75 commented Jan 20, 2022

layday commented Jan 20, 2022 • edited Loading

henryiii commented Jan 20, 2022 • edited Loading

FFY00 commented Jan 20, 2022

hroncok commented Jan 31, 2022

hroncok commented Jan 31, 2022

layday commented Jan 31, 2022 via email

linsomniac commented May 19, 2022

petersilva commented Dec 2, 2022

henryiii commented Dec 2, 2022 • edited Loading

henryiii commented Dec 2, 2022 • edited Loading

henryiii commented Dec 2, 2022

zoot-inge commented Sep 1, 2023

nschloe commented Jan 28, 2021 •

edited

Loading

layday commented Jan 19, 2022 •

edited

Loading

layday commented Jan 19, 2022 •

edited

Loading

layday commented Jan 20, 2022 •

edited

Loading

henryiii commented Jan 20, 2022 •

edited

Loading

henryiii commented Dec 2, 2022 •

edited

Loading

henryiii commented Dec 2, 2022 •

edited

Loading