poetry install does not honor build-system.requires

[sharvil]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 20.04.4
• Poetry version: 1.2.0b2
• Link of a Gist with the contents of your pyproject.toml file:

pyproject.toml

[tool.poetry]
name = "meow"
version = "0.1.0"
description = ""
authors = [""]
build = "build.py"

[tool.poetry.dependencies]
python = "^3.9"
SoundFile = "0.10.*"
crcmod = "^1.7"
librosa = ">=0.8.0"
numba = "^0.56.0"
protobuf = "3.14.0"
tensorboard = "^2.10.0"
torchaudio = {version = "=0.12.1+cu113", source="torch"}
torch = {version = "=1.12.1+cu113", source="torch"}

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu113"
secondary = true

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0", "setuptools>=40.6.0", "wheel", "numpy", "pybind11"]
build-backend = "poetry.core.masonry.api"

Issue

Running poetry build with the above pyproject.toml does the right thing:

Preparing build environment with build-system requirements poetry-core>=1.0.0, setuptools>=40.6.0, wheel, numpy, pybind11
[...snip...]

Running poetry install fails because it attempts to build the project without installing build-system.requires packages.

I would have expected the install command to create an isolated environment, install the build requirements and perform the build in that isolated environment, and then install the produced artifact in my current environment.

[bilelomrani1]

Same issue on macOS 12.5. A dirty hack is to add the following at the top of build.py:

import subprocess

try:
    import pybind11
except ImportError:
    subprocess.run(["pip", "install", "pybind11"])

The exact same issue was described in #2789, which was marked as solved, but the issue is still present on Poetry 1.2.0b3. I didn't check if this is a regression from when this PR was merged.

[neersighted]

This is currently intentional -- Poetry has never respected the build-system entry for a Poetry-managed project. We might want to establish consensus on this being a desired behavior, since currently you can install any needed build dependencies using a Poetry group or similar (but this will result in duplication of deps in pyproject.toml).

[neersighted]

Re: this being a regression, the changes in that PR were explicitly scoped to use of build scripts.

Also, Poetry does not perform builds in an isolated/pep 517 environment, but instead uses the 'main' environment poetry install uses, as this is the only way to use lockfile versions during the build with the current architecture. It is possible to re-architect this so that a clean build environment is used and the build-system deps are only installed there (as opposed to polluting the dev group implicitly or similar) -- however that will require the implementation of a lockfile-aware poetry-core first (which we have already discussed/planned).

[ghost]

Is this the same issue as [tool.poetry.dependencies] deps being unavailable to tool.poetry.build.script in 1.2.0 despite being installed?

[arcbtc]

+1, having exact same issue tool.poetry.build.script is not using dependencies from [tool.poetry.dependencies]

[randolf-scholz]

@neersighted For me this issue occurs when I have a GIT-dependency that has jupyter-packaging as a build dependency. (specifically this fork here: https://github.com/randolf-scholz/jupyterlab_templates)

Also, it does seem poetry uses an isolated environment to build this package.

Command ['/tmp/tmpx6y_j1ku/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output: 
Traceback (most recent call last):
  File "/home/rscholz/Projects/KIWI/python-venv/.venv/src/jupyterlab_templates/setup.py", line 4, in <module>
    from jupyter_packaging import (
ModuleNotFoundError: No module named 'jupyter_packaging'

`/tmp/tmpx6y_j1ku/.venv/bin/pip list`

Package    Version
---------- -------
build      0.7.0
packaging  21.3
pep517     0.12.0
pip        22.3.1
pyparsing  3.0.9
setuptools 65.5.1
tomli      2.0.1
wheel      0.37.1

[neersighted]

Your issue is entirely different @randolf-scholz -- this issue is about build-system.requires for Poetry-managed projects, and not for dependencies that don't use Poetry. Please reach out via Discord or Discussions for support.

[adamgreig]

I believe @randolf-scholz's issue is what I've reported as a separate issue in #7702, I couldn't find any other issue about it.

[Samreay]

I raised this at #7909 as a potentially separate issue, but it might be the same thing I'm encountering.

The pyproject.toml is available on the linked issue, but for a TLDR here, we have a build.py that compiles cython, and so our build requires include both cython and numpy. A poetry build works fine. But a poetry install does not, because it seems when "Getting requirements to build editable", setuptools/build_meta.py is run (via the --isolated tag in the parent pip invocation) in an environment which does not respect the build requirements.

Full stack from our failing docker build below:

=> [builder 11/12] RUN rm -f /app/id_rsa && poetry build                                                                                                           17.1s
 => ERROR [builder 12/12] RUN poetry install --only-root -vvv                                                                                                        3.7s
------
 > [builder 12/12] RUN poetry install --only-root -vvv:
#0 0.784 Loading configuration file /app/poetry.toml
#0 0.818 Skipping virtualenv creation, as specified in config file.
#0 0.998 Installing dependencies from lock file
#0 1.181 
#0 1.181 Finding the necessary packages for the current system
#0 1.195 
#0 1.195 Installing the current project: optimiser (0.1.0)
#0 1.195   - Building package optimiser in editable mode
#0 1.195   - Falling back on using a setup.py
#0 3.141 
#0 3.141   Stack trace:
#0 3.141 
#0 3.141   2  /opt/venv/lib/python3.9/site-packages/poetry/utils/env.py:1543 in _run
#0 3.206       1541│                 return subprocess.call(cmd, stderr=stderr, env=env, **kwargs)
#0 3.206       1542│             else:
#0 3.206     → 1543│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
#0 3.207       1544│         except CalledProcessError as e:
#0 3.207       1545│             raise EnvCommandError(e, input=input_)
#0 3.207 
#0 3.207   1  /usr/local/lib/python3.9/subprocess.py:424 in check_output
#0 3.261        422│         kwargs['input'] = empty
#0 3.261        423│ 
#0 3.261     →  424│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#0 3.261        425│                **kwargs).stdout
#0 3.261        426│ 
#0 3.261 
#0 3.261   CalledProcessError
#0 3.261 
#0 3.261   Command '['/opt/venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/opt/venv', '--upgrade', '--no-deps', '-e', '/app']' returned non-zero exit status 1.
#0 3.261 
#0 3.261   at /usr/local/lib/python3.9/subprocess.py:528 in run
#0 3.317        524│             # We don't call process.wait() as .__exit__ does that for us.
#0 3.317        525│             raise
#0 3.317        526│         retcode = process.poll()
#0 3.317        527│         if check and retcode:
#0 3.317     →  528│             raise CalledProcessError(retcode, process.args,
#0 3.317        529│                                      output=stdout, stderr=stderr)
#0 3.317        530│     return CompletedProcess(process.args, retcode, stdout, stderr)
#0 3.317        531│ 
#0 3.317        532│ 
#0 3.317 
#0 3.317 The following error occurred when trying to handle this error:
#0 3.317 
#0 3.317 
#0 3.317   Stack trace:
#0 3.317 
#0 3.317   2  /opt/venv/lib/python3.9/site-packages/poetry/utils/pip.py:56 in pip_install
#0 3.319        54│ 
#0 3.319        55│     try:
#0 3.319     →  56│         return environment.run_pip(*args)
#0 3.319        57│     except EnvCommandError as e:
#0 3.319        58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#0 3.319 
#0 3.319   1  /opt/venv/lib/python3.9/site-packages/poetry/utils/env.py:1505 in run_pip
#0 3.379       1503│         pip = self.get_pip_command()
#0 3.379       1504│         cmd = pip + list(args)
#0 3.379     → 1505│         return self._run(cmd, **kwargs)
#0 3.379       1506│ 
#0 3.379       1507│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:
#0 3.379 
#0 3.379   EnvCommandError
#0 3.380 
#0 3.380   Command ['/opt/venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/opt/venv', '--upgrade', '--no-deps', '-e', '/app'] errored with the following return code 1
#0 3.380   
#0 3.380   Output:
#0 3.380   Obtaining file:///app
#0 3.380     Installing build dependencies: started
#0 3.380     Installing build dependencies: finished with status 'done'
#0 3.380     Checking if build backend supports build_editable: started
#0 3.380     Checking if build backend supports build_editable: finished with status 'done'
#0 3.380     Getting requirements to build editable: started
#0 3.380     Getting requirements to build editable: finished with status 'error'
#0 3.380     error: subprocess-exited-with-error
#0 3.380     
#0 3.380     × Getting requirements to build editable did not run successfully.
#0 3.380     │ exit code: 1
#0 3.380     ╰─> [21 lines of output]
#0 3.380         Traceback (most recent call last):
#0 3.380           File "/opt/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#0 3.380             main()
#0 3.380           File "/opt/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#0 3.380             json_out['return_val'] = hook(**hook_input['kwargs'])
#0 3.380           File "/opt/venv/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
#0 3.380             return hook(config_settings)
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 450, in get_requires_for_build_editable
#0 3.380             return self.get_requires_for_build_wheel(config_settings)
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
#0 3.380             return self._get_build_requires(config_settings, requirements=['wheel'])
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
#0 3.380             self.run_setup()
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
#0 3.380             super(_BuildMetaLegacyBackend,
#0 3.380           File "/tmp/pip-build-env-gt_yalkq/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
#0 3.380             exec(code, locals())
#0 3.380           File "<string>", line 48, in <module>
#0 3.380           File "/app/builder.py", line 3, in <module>
#0 3.380             import numpy as np
#0 3.380         ModuleNotFoundError: No module named 'numpy'
#0 3.380         [end of output]
#0 3.380     
#0 3.380     note: This error originates from a subprocess, and is likely not a problem with pip.
#0 3.380   error: subprocess-exited-with-error
#0 3.380   
#0 3.380   × Getting requirements to build editable did not run successfully.
#0 3.380   │ exit code: 1
#0 3.380   ╰─> See above for output.
#0 3.380   
#0 3.380   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 3.380   
#0 3.380 
#0 3.380   at /opt/venv/lib/python3.9/site-packages/poetry/utils/env.py:1545 in _run
#0 3.440       1541│                 return subprocess.call(cmd, stderr=stderr, env=env, **kwargs)
#0 3.440       1542│             else:
#0 3.440       1543│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
#0 3.440       1544│         except CalledProcessError as e:
#0 3.440     → 1545│             raise EnvCommandError(e, input=input_)
#0 3.440       1546│ 
#0 3.441       1547│         return decode(output)
#0 3.441       1548│ 
#0 3.441       1549│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
#0 3.441 
#0 3.441 The following error occurred when trying to handle this error:
#0 3.441 
#0 3.441 
#0 3.441   Stack trace:
#0 3.441 
#0 3.441   10  /opt/venv/lib/python3.9/site-packages/cleo/application.py:327 in run
#0 3.460        325│ 
#0 3.460        326│             try:
#0 3.460      → 327│                 exit_code = self._run(io)
#0 3.460        328│             except BrokenPipeError:
#0 3.460        329│                 # If we are piped to another process, it may close early and send a
#0 3.460 
#0 3.460    9  /opt/venv/lib/python3.9/site-packages/poetry/console/application.py:190 in _run
#0 3.473        188│         self._load_plugins(io)
#0 3.473        189│ 
#0 3.473      → 190│         exit_code: int = super()._run(io)
#0 3.473        191│         return exit_code
#0 3.473        192│ 
#0 3.473 
#0 3.473    8  /opt/venv/lib/python3.9/site-packages/cleo/application.py:431 in _run
#0 3.492        429│             io.input.interactive(interactive)
#0 3.492        430│ 
#0 3.492      → 431│         exit_code = self._run_command(command, io)
#0 3.492        432│         self._running_command = None
#0 3.492        433│ 
#0 3.492 
#0 3.492    7  /opt/venv/lib/python3.9/site-packages/cleo/application.py:473 in _run_command
#0 3.512        471│ 
#0 3.512        472│         if error is not None:
#0 3.512      → 473│             raise error
#0 3.512        474│ 
#0 3.512        475│         return terminate_event.exit_code
#0 3.512 
#0 3.512    6  /opt/venv/lib/python3.9/site-packages/cleo/application.py:457 in _run_command
#0 3.532        455│ 
#0 3.532        456│             if command_event.command_should_run():
#0 3.532      → 457│                 exit_code = command.run(io)
#0 3.532        458│             else:
#0 3.532        459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
#0 3.532 
#0 3.532    5  /opt/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
#0 3.537        117│         io.input.validate()
#0 3.537        118│ 
#0 3.537      → 119│         status_code = self.execute(io)
#0 3.537        120│ 
#0 3.537        121│         if status_code is None:
#0 3.537 
#0 3.537    4  /opt/venv/lib/python3.9/site-packages/cleo/commands/command.py:62 in execute
#0 3.546         60│ 
#0 3.546         61│         try:
#0 3.546      →  62│             return self.handle()
#0 3.546         63│         except KeyboardInterrupt:
#0 3.546         64│             return 1
#0 3.546 
#0 3.546    3  /opt/venv/lib/python3.9/site-packages/poetry/console/commands/install.py:187 in handle
#0 3.551        185│             return 0
#0 3.551        186│ 
#0 3.551      → 187│         builder.build()
#0 3.551        188│ 
#0 3.551        189│         if overwrite:
#0 3.551 
#0 3.551    2  /opt/venv/lib/python3.9/site-packages/poetry/masonry/builders/editable.py:63 in build
#0 3.560         61│                     "  - Falling back on using a setup.py"
#0 3.560         62│                 )
#0 3.560      →  63│                 self._setup_build()
#0 3.560         64│                 path: Path = self._path
#0 3.560         65│                 return path
#0 3.560 
#0 3.560    1  /opt/venv/lib/python3.9/site-packages/poetry/masonry/builders/editable.py:112 in _setup_build
#0 3.569        110│                 )
#0 3.569        111│                 try:
#0 3.569      → 112│                     pip_install(self._path, self._env, upgrade=True, editable=True)
#0 3.569        113│                 finally:
#0 3.569        114│                     shutil.move(
#0 3.569 
#0 3.569   PoetryException
#0 3.569 
#0 3.569   Failed to install /app
#0 3.569 
#0 3.569   at /opt/venv/lib/python3.9/site-packages/poetry/utils/pip.py:58 in pip_install
#0 3.570        54│ 
#0 3.571        55│     try:
#0 3.571        56│         return environment.run_pip(*args)
#0 3.571        57│     except EnvCommandError as e:
#0 3.571     →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
#0 3.571        59│ 
------
Dockerfile:26
--------------------
  24 |     WORKDIR /app
  25 |     RUN rm -f /app/id_rsa && poetry build 
  26 | >>> RUN poetry install --only-root -vvv
  27 |     
  28 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c poetry install --only-root -vvv" did not complete successfully: exit code: 1
make: *** [Makefile:21: build_image] Error 1

[radoering]

Trying to clarify some statements:

The exact same issue was described in #2789, which was marked as solved, but the issue is still present on Poetry 1.2.0b3. I didn't check if this is a regression from when #5401 was merged.

Not the same issue, only similar. #2789 descibes the use of a build script with generate-setup-file = false and #5401 fixed exactly that issue. However, this issue describes the use of a build script with generate-setup-file = true.

This is currently intentional -- Poetry has never respected the build-system entry for a Poetry-managed project. We might want to establish consensus on this being a desired behavior, since currently you can install any needed build dependencies using a Poetry group or similar (but this will result in duplication of deps in pyproject.toml).

Also, Poetry does not perform builds in an isolated/pep 517 environment, but instead uses the 'main' environment poetry install uses, as this is the only way to use lockfile versions during the build with the current architecture.

Maybe, that was true in the past, but it's not anymore for all kinds of Poetry-managed projects (at least since #5401). I think it's still true for projects without build scripts but in this case there are no build-system requirements except for poetry-core anyway.

As it seems projects with build scripts are built in an isolated environment (no matter if generate-setup-file is true of false).

Is this the same issue as [tool.poetry.dependencies] deps being unavailable to tool.poetry.build.script in 1.2.0 despite being installed?

Not the same, but it shows that neither locked dependencies nor build-system requirements are available. Locked dependencies are not available because an isolated environment is used instead of the project's environment.

#7975 looks like a sensible fix. It will make build-system requirements available.

It might still be surprising that locked dependencies are not available when building projects with build scripts but that's really something were consensus is needed (and that's not what this issue actually was about in the first place).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.