Mysterious error with optional dependencies in pyproject.toml

[alexdewar]

I have been following the guide in the README file on how to use pip-tools with pyproject.toml. Unfortunately, if I add a project.optional-dependencies section to pyproject.toml, regardless of what its contents are, I get the following error:

(base) PS C:\Users\adewar\code\Solidity-GUI> pip-compile -v pyproject.toml
C:\ProgramData\Miniconda3\lib\site-packages\_distutils_hack\__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Creating virtualenv isolated environment...
find interpreter for spec PythonSpec(path=C:\ProgramData\Miniconda3\python.exe)
proposed PythonInfo(spec=CPython3.9.13.final.0-64, exe=C:\ProgramData\Miniconda3\python.exe, platform=win32, version='3.9.13 (main, Oct 13 2022, 21:23:06) [MSC v.1916 64 bit (AMD64)]', encoding_fs_io=utf-8-utf-8)
create virtual environment via CPython3Windows(dest=C:\Users\adewar\AppData\Local\Temp\build-env-m1awcc93, clear=False, no_vcs_ignore=False, global=False)
add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=C:\Users\adewar\AppData\Local\pypa\virtualenv)
Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse C:\Users\adewar\code\Solidity-GUI\pyproject.toml

Environment Versions

1. Windows 10 (replicated on Arch Linux)
2. Python version: python 3.9.13
3. pip version: pip 22.2.2 from C:\ProgramData\Miniconda3\lib\site-packages\pip (python 3.9)
4. pip-tools version: pip-compile, version 6.9.0

Steps to replicate

1. Add a project.optional-dependencies section to your pyproject.toml
2. Run pip-compile pyproject.toml

[alexdewar]

I've dug in a bit more and it's weirder than I thought. This following pyproject.toml doesn't work:

[project]
name = "SolidityGUI"
version = "0.1"
authors = ["author1"]
dependencies = ["pygmsh", "pyside6", "vtk"]

[project.optional-dependencies]
dev = ["black"]

However, if you remove the authors line, it works fine again:

[project]
name = "SolidityGUI"
version = "0.1"
dependencies = ["pygmsh", "pyside6", "vtk"]

[project.optional-dependencies]
dev = ["black"]

How odd is that?!

[AndydeCleyre]

Thanks! Can you post the entire pyproject.toml, including the build-system section?

[alexdewar]

That is the entire pyproject.toml. Is the build-system section a requirement that I'm missing...?

[AndydeCleyre]

It usually should be there if it's describing an installable package, but I'm not sure if it ought to be strictly mandatory here. Looking at PEP 518:

For the vast majority of Python projects that rely upon setuptools, the pyproject.toml file will be:

[build-system]
# Minimum requirements for the build system to execute.
requires = ["setuptools", "wheel"]  # PEP 508 specifications.

...

Tools should not require the existence of the [build-system] table. A pyproject.toml file may be used to store configuration details other than build-related data and thus lack a [build-system] table legitimately. If the file exists but is lacking the [build-system] table then the default values as specified above should be used. If the table is specified but is missing required fields then the tool should consider it an error.

[AndydeCleyre]

My thought at the moment is that if you add the above sample section and it works, then this is a bug, since when omitted

. . . the default values as specified above should be used.

[alexdewar]

A colleague has tracked down the source of the problem. If you check pyproject.toml with validate-pyproject you get:

[ERROR] project.authors[{data__authors_x}] must be object

Putting the authors field into the correct format fixes things, i.e.:

authors = [{ name = "author1", email = "author1@email.com" }]

[atugushev]

I'm glad you've found the root of the issue. I'll close this since there's no action required regarding pip-tools. Thanks for the issue!

[alexdewar]

Thanks for your help @atugushev. It may still be worth opening a separate issue re having better error reporting in the case where the pyproject.toml file is malformed, but obviously it's a less important problem.

[atugushev]

$ pip-compile pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /private/var/folders/l0/lnq1ghps5yqc4vkgszlcz92m0000gp/T/tmp.X3gWYzsJ/foo/pyproject.toml

If you think that this could be improved on the build side feel free to report on https://github.com/pypa/build/issues.

[atugushev]

Ouch, sorry. This is the pip-tools' message:

pip-tools/piptools/scripts/compile.py

Line 479 in eff8476

log.error(f"Failed to parse {os.path.abspath(src_file)}")

---

pip install .:

      ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
      configuration error: `project.authors[{data__authors_x}]` must be object

---

python -m build:

ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
configuration error: `project.authors[{data__authors_x}]` must be object

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

[vduseev]

I've stumbled upon this issue a couple of times in the last month. Here is the error I see:

$ pip-compile pyproject.toml -v
Creating venv isolated environment...
Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/OSS/opensearch-logger/pyproject.toml

Setup

Environment:

• Python 3.11
• MacOS 13.0.1
• arm64

The pyproject.toml file itself is super simple and passes validation using the validate-pyproject package:

[build-system]
requires = [
    "flit_core >=3.2,<4",
]
build-backend = "flit_core.buildapi"

[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]

What causes it

It's definitely not a pip-compile problem as it just calls project_wheel_metadata from the build package and then catches the error and reports.

pip-tools/piptools/scripts/compile.py

Lines 474 to 476 in eff8476

	metadata = project_wheel_metadata(
	os.path.dirname(os.path.abspath(src_file))
	)

The build package, in turn, does lots of stuff and ends up calling a Python script called _in_process.py from the pep517 package.

I have no idea how it is supposed to work, but it looks like the build package create a one-time temporary virtual environment in which this _in_process.py script is then invoked. Here is the command with which build package tries to call a subprocess in my case:

['/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-rj6bf2id/bin/python', '/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpb6wdunjz']

Obviously, when something wrong happens in that subprocess script all we get at the end is:

Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/Playground/pip-tools-bug/pyproject.toml

But in reality, the real error does not ever get propagated to the parent process and we don't see it. I've caught it by instrumenting the _in_process.py file in my virtual environment (lib/python3.11/site-packages/pep517/in_process/_in_process.py) with print statements.

Exception: description must be specified under [project] or listed as a dynamic field
_in_process.py was called with args: ['/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpttsp4wmf']
Exception: description must be specified under [project] or listed as a dynamic field
  File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 338, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/buildapi.py", line 23, in get_requires_for_build_wheel
    info = read_flit_config(pyproj_toml)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 79, in read_flit_config
    return prep_toml_config(d, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 106, in prep_toml_config
    loaded_cfg = read_pep621_metadata(d['project'], path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
    raise ConfigError(

Now I can see that the underlying error actually comes from the flit package, complaining about lack of the description field in the [project] section of my pyproject.toml file.

How to get the same output

Here is what I manually added to the _in_process.py file in pep517 package in my venv

https://github.com/pypa/pyproject-hooks/blob/4c9325f4e594bfc7178452d0d01eb8da6c3dbbdb/src/pyproject_hooks/_in_process/_in_process.py#L322

def main():
    # begin: added
    from pathlib import Path
    log_file = Path(__file__).parent / "log.txt"
    with log_file.open("a") as f:
        try:
    # end: added
            if len(sys.argv) < 3:
                sys.exit("Needs args: hook_name, control_dir")
            hook_name = sys.argv[1]
            control_dir = sys.argv[2]
            if hook_name not in HOOK_NAMES:
                sys.exit("Unknown hook: %s" % hook_name)
            hook = globals()[hook_name]

            hook_input = read_json(pjoin(control_dir, 'input.json'))

            json_out = {'unsupported': False, 'return_val': None}
            try:
                json_out['return_val'] = hook(**hook_input['kwargs'])
            except BackendUnavailable as e:
                json_out['no_backend'] = True
                json_out['traceback'] = e.traceback
            except BackendInvalid as e:
                json_out['backend_invalid'] = True
                json_out['backend_error'] = e.message
            except GotUnsupportedOperation as e:
                json_out['unsupported'] = True
                json_out['traceback'] = e.traceback
            except HookMissing as e:
                json_out['hook_missing'] = True
                json_out['missing_hook_name'] = e.hook_name or hook_name

            write_json(json_out, pjoin(control_dir, 'output.json'), indent=2)
        # begin: more_added
        except Exception as e:
            f.write(f"Exception: {e}\n")
            import traceback
            traceback.print_tb(e.__traceback__, file=f)
        # end: more_added

Fixing the issue

I can address the complaints of flit by adding missing things:

• Adding description field
• Making sure a module opensearch_logger exists in the same directory where pyproject.toml is placed

And voila:

#
# This file is autogenerated by pip-compile with python 3.11
# To update, run:
#
#    pip-compile pyproject.toml
#
certifi==2022.9.24
    # via
    #   opensearch-py
    #   requests
charset-normalizer==2.1.1
    # via requests
idna==3.4
    # via requests
opensearch-py==2.0.0
    # via opensearch-logger (pyproject.toml)
requests==2.28.1
    # via opensearch-py
urllib3==1.26.13
    # via
    #   opensearch-py
    #   requests

Alternatively, should I swap the build system to classic setuptools, it compiles successfully even without said fixes.

[build-system]
requires = [
    "setuptools",
    "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]

Conclusion

I don't understand why such system was chosen by creators of build system (not pip-tools). This is the reason we don't see an underlying problem from the chosen build system. It just does not get propagated to higher level tools, such as build or pip-tools.

P.S. was so happy to see @atugushev in the comments of this issue. Cheers!

[atugushev]

@vduseev thanks for chiming in and for the additional context!

Digging into the issue I found that build.utils.project_wheel_metadata uses pep517.quiet_subprocess_runner which suppresses output. With pep517.default_subprocess_runner It shows the actual error.

Patched build:

diff --git src/build/util.py src/build/util.py
index 9675393..a435479 100644
--- src/build/util.py
+++ src/build/util.py
@@ -43,7 +43,7 @@ def project_wheel_metadata(
     """
     builder = build.ProjectBuilder(
         os.fspath(srcdir),
-        runner=pep517.quiet_subprocess_runner,
+        runner=pep517.default_subprocess_runner,
     )

     if not isolated:

pip-compile output:

root@35878ab01f42:/foo# pip-compile pyproject.toml
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/usr/local/lib/python3.10/site-packages/flit_core/buildapi.py", line 47, in prepare_metadata_for_build_wheel
    ini_info = read_flit_config(pyproj_toml)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 79, in read_flit_config
    return prep_toml_config(d, path)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 106, in prep_toml_config
    loaded_cfg = read_pep621_metadata(d['project'], path)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
    raise ConfigError(
flit_core.config.ConfigError: description must be specified under [project] or listed as a dynamic field
Backend subprocess exited when trying to invoke prepare_metadata_for_build_wheel
Failed to parse /foo/pyproject.toml

---

@pradyunsg I wonder if would it be okay to introduce one of the following params in build.utils. project_wheel_metadata()? For example:

• project_wheel_metadata(quiet=False) - do not suppress the output
• project_wheel_metadata(runner=my_runner) - use my runner

[atugushev]

FTR: tracking issue pypa/build#553 for project_wheel_metadata improvement.

[pradyunsg]

@pradyunsg I wonder if would it be okay to introduce one of the following params in build.utils. project_wheel_metadata()?

I just saw this, and I would've suggested that you do exactly what you've done. :)

[jasonsketchup]

I had a bad whitespace, a '\t' (tab) in a quoted string, that caused this issue; TOML Linter found it.

[cout]

I found another case today where this error can occur:. I started with a working pyproject.toml, then I created two subdirectories.

The clue for me was that pip install --editable . gave an error "multiple top-level packages discovered in a flat-layout".

Running validate-projected returned no errors.

$ cat pyproject.toml 
[project]
name = 'test'
description = 'test'
version = '0.1.0'
authors = [ ]
dependencies = [ 'simplejson ' ]
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
    # via test (pyproject.toml)
$ mkdir foo
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
    # via test (pyproject.toml)
$ mkdir bar
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /home/pbrannan/tmp/test-env/pyproject.toml
$ ./env/bin/python -mvalidate_pyproject pyproject.toml
Valid file: pyproject.toml

[AndydeCleyre]

Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with

[tool.setuptools]
py-modules = []

---

Aside from that I'll note that the alternative runner selection is now in build, but there has not yet been a release including it.

[vietvudanh]

pip-compile omitted all error logs, I have to use pip install -e . to see what went wrong with the pyproject.toml file. My problem was wrong syntax for fields license and requires-python

[engineervix]

Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with

[tool.setuptools]
py-modules = []

Aside from that I'll note that the alternative runner selection is now in build, but there has not yet been a release including it.

I also experienced this problem. I have a valid pyproject.toml with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre's solution above is what worked for me. Thanks @AndydeCleyre!

[vlad-ivanov-name]

for the record, build has implemented switching runner from the default quiet one, but it's not yet included in any releases of build.

[dre-hh]

So the errors come because pip-compile swallows output of pip install -e .
Running the command will reveal the error.

In the case of a flatlayout with multiple packages, mentioned by @cout
one can explicitly specify which folders in the flat layout must be included as packages

[tool.setuptools]
packages = ["package1", "package2"]

[khimaros]

Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with

[tool.setuptools]
py-modules = []

Aside from that I'll note that the alternative runner selection is now in build, but there has not yet been a release including it.

I also experienced this problem. I have a valid pyproject.toml with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre's solution above is what worked for me. Thanks @AndydeCleyre!

this also solved the problem for me.

[Opalo]

python -m build

Thanks for that, gives way more descriptive error and it turned out that it my case it was related to write permissions.

[lafrech]

I had the same error

Backend subprocess exited when trying to invoke get_requires_for_build_wheel

validate-pyproject wouldn't yield any issue and the runner=pep517.default_subprocess_runner trick didn't help.

I rolled back pip-tools to 7.3.0 then for some reason I got a more useful output:

ValueError: Multiple files or folders could be module my_module

Because I had moved my source files from project root to a src folder and after a bit of back and forth in git, the old folder had reappeared empty.

I figured I'd leave a not here in case it helps anyone.