CLI tools defined in 'build_requires' not available during dependency install

[greschd]

• Poetry version: 1.7.0
• Python version: 3.10
• OS version and name: Debian GNU/Linux 12 (bookworm)
• pyproject.toml: https://github.com/greschd/poetry_bug_repro/blob/main/poetry_project/pyproject.toml

Issue

When installing a poetry project with dependencies which require a CLI tool (defined in their pyproject.toml [build-system]requires section), the build fails since the CLI tool is not available when the dependency is built. Building the dependency directly using python -m build works.

The following repository contains a minimal reproduction: https://github.com/greschd/poetry_bug_repro The CLI tool chosen is pre-commit==2.3.0. [1]

Steps to reproduce:

Option 1 (using Docker):

• git clone https://github.com/greschd/poetry_bug_repro.git
• cd poetry_bug_repro
• docker build .

This builds the failing_build target, which errors out with the traceback below (pre-commit does not exist). To see that python -m build works (in the same directory):

• docker build --target=working_build .

Option 2 (direct):

• git clone https://github.com/greschd/poetry_bug_repro.git
• cd poetry_bug_repro/poetry_project
• poetry install

This will either produce the same traceback as below, or complain that the pre-commit version is wrong (if pre-commit is available in the wider environment).

Full log:

 > [failing_build 6/6] RUN poetry install -vvv:                                                                                                                                                                                          
1.220 Creating virtualenv poetry-project-2nQEdy2R-py3.10 in /root/.cache/pypoetry/virtualenvs                                                                                                                                            
1.552 Using virtualenv: /root/.cache/pypoetry/virtualenvs/poetry-project-2nQEdy2R-py3.10                                                                                                                                                 
1.689 Installing dependencies from lock file                                                                                                                                                                                             
1.692                                                                                                                                                                                                                                    
1.692 Finding the necessary packages for the current system
1.697 
1.697 Package operations: 1 install, 0 updates, 0 removals
1.697 
1.698   • Installing dependency-project (0.2.0 /dependency_project)
6.914 
6.914   Stack trace:
6.914 
6.914   6  /usr/local/lib/python3.10/site-packages/poetry/installation/executor.py:269 in _execute_operation
6.958       267│ 
6.958       268│             try:
6.958     → 269│                 result = self._do_execute_operation(operation)
6.958       270│             except EnvCommandError as e:
6.958       271│                 if e.e.returncode == -2:
6.958 
6.959   5  /usr/local/lib/python3.10/site-packages/poetry/installation/executor.py:379 in _do_execute_operation
6.992       377│             return 0
6.992       378│ 
6.992     → 379│         result: int = getattr(self, f"_execute_{method}")(operation)
6.992       380│ 
6.992       381│         if result != 0:
6.992 
6.992   4  /usr/local/lib/python3.10/site-packages/poetry/installation/executor.py:504 in _execute_install
7.028       502│ 
7.028       503│     def _execute_install(self, operation: Install | Update) -> int:
7.028     → 504│         status_code = self._install(operation)
7.028       505│ 
7.028       506│         self._save_url_reference(operation)
7.028 
7.029   3  /usr/local/lib/python3.10/site-packages/poetry/installation/executor.py:536 in _install
7.063       534│             archive = self._prepare_archive(operation)
7.063       535│         elif package.source_type == "directory":
7.063     → 536│             archive = self._prepare_archive(operation)
7.063       537│             cleanup_archive = True
7.063       538│         elif package.source_type == "url":
7.063 
7.063   2  /usr/local/lib/python3.10/site-packages/poetry/installation/executor.py:608 in _prepare_archive
7.097       606│         self._populate_hashes_dict(archive, package)
7.097       607│ 
7.097     → 608│         return self._chef.prepare(
7.097       609│             archive, editable=package.develop, output_dir=output_dir
7.097       610│         )
7.097 
7.098   1  /usr/local/lib/python3.10/site-packages/poetry/installation/chef.py:121 in prepare
7.109       119│         if archive.is_dir():
7.109       120│             destination = output_dir or Path(tempfile.mkdtemp(prefix="poetry-chef-"))
7.110     → 121│             return self._prepare(archive, destination=destination, editable=editable)
7.110       122│ 
7.110       123│         return self._prepare_sdist(archive, destination=output_dir)
7.110 
7.110   ChefBuildError
7.110 
7.110   Backend subprocess exited when trying to invoke get_requires_for_build_wheel
7.110   
7.110   Traceback (most recent call last):
7.110     File "/usr/local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
7.110       main()
7.110     File "/usr/local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
7.110       json_out['return_val'] = hook(**hook_input['kwargs'])
7.110     File "/usr/local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
7.110       return hook(config_settings)
7.110     File "/tmp/tmps0obq4kj/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
7.110       return self._get_build_requires(config_settings, requirements=['wheel'])
7.110     File "/tmp/tmps0obq4kj/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
7.110       self.run_setup()
7.110     File "/tmp/tmps0obq4kj/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 507, in run_setup
7.110       super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
7.110     File "/tmp/tmps0obq4kj/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
7.110       exec(code, locals())
7.110     File "<string>", line 5, in <module>
7.110     File "/usr/local/lib/python3.10/subprocess.py", line 421, in check_output
7.110       return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
7.110     File "/usr/local/lib/python3.10/subprocess.py", line 503, in run
7.110       with Popen(*popenargs, **kwargs) as process:
7.110     File "/usr/local/lib/python3.10/subprocess.py", line 971, in __init__
7.110       self._execute_child(args, executable, preexec_fn, close_fds,
7.110     File "/usr/local/lib/python3.10/subprocess.py", line 1863, in _execute_child
7.110       raise child_exception_type(errno_num, err_msg, err_filename)
7.110   FileNotFoundError: [Errno 2] No such file or directory: 'pre-commit'
7.110   
7.110 
7.110   at /usr/local/lib/python3.10/site-packages/poetry/installation/chef.py:166 in _prepare
7.122       162│ 
7.122       163│                 error = ChefBuildError("\n\n".join(message_parts))
7.122       164│ 
7.122       165│             if error is not None:
7.122     → 166│                 raise error from None
7.122       167│ 
7.122       168│             return path
7.122       169│ 
7.122       170│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
7.123 
7.123 Note: This error originates from the build backend, and is likely not a problem with poetry but with dependency-project (0.2.0 /dependency_project) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "dependency-project @ file:///dependency_project"'.
7.123 

[1] This is only for demonstration purposes, it could be any CLI tool. A somewhat old version of pre-commit is used to make it less likely to already exist in the environment.

[dimbleby]

diff --git a/src/poetry/installation/chef.py b/src/poetry/installation/chef.py
index d801fb6d..4d554696 100644
--- a/src/poetry/installation/chef.py
+++ b/src/poetry/installation/chef.py
@@ -129,10 +129,8 @@ class Chef:

         with ephemeral_environment(self._env.python) as venv:
             env = IsolatedEnv(venv, self._pool)
-            builder = ProjectBuilder(
-                directory,
-                python_executable=env.python_executable,
-                runner=quiet_subprocess_runner,
+            builder = ProjectBuilder.from_isolated_env(
+                env, directory, runner=quiet_subprocess_runner
             )
             env.install(builder.build_system_requires)

should do it

(else build does not call the "wrapped" subprocess runner that includes make_extra_environ())

[OlivierLDff]

I can observe this in my project that uses maturin. I guess it can be reproduced with the default maturin project.

[dimbleby]

pull request welcome, I'm sure

[radoering]

Since it hasn't been mentioned yet: This issue is a regression from 1.6 to 1.7 caused by the transition from build 0.10 to build 1.0 (#8400). Fix: #8630

[greschd]

Thanks for the quick feedback, and getting a fix ready 🎉 I can confirm that this is fixed with the PR branch.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.