pyproject.toml (PEP518), build isolation (PEP517) and optional build dependencies

[paugier]

What's the problem this feature will solve?

Projects that have build requirements and optional build requirements can not use pyproject.toml because of the build isolation.

Example (https://bitbucket.org/fluiddyn/fluidfft): depending if mpi4py is installed, MPI extensions are built or not. It's very important to be able to install/use the package without MPI so mpi4py can not be added in build-system.requires.

If a pyproject.toml is used, we get ImportError when importing mpi4py in the setup.py even when mpi4py is installed because of the isolation! So the MPI extensions are never build.

Describe the solution you'd like

Something like this could work:

[build-system]
# Minimum requirements for the build system to execute.
requires = [
    "setuptools", "wheel", "numpy", "cython", "jinja2", "fluidpythran>=0.1.7"
]
# packages that are used during the build only if they are installed
optional = ["mpi4py", "pythran"]

Then mpi4py would be available during the build only if it is installed. We keep the advantages of the isolation (discussed in https://www.python.org/dev/peps/pep-0517/) but optional build dependencies are allowed.

Alternative Solutions

[build-system]
# Minimum requirements for the build system to execute.
requires = [
    "setuptools", "wheel", "numpy", "cython", "jinja2", "fluidpythran>=0.1.7"
]
isolation = False

[pradyunsg]

You can use the --no-build-isolation flag.

[paugier]

From the help message "--no-build-isolation Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is used." i don't think it is a solution !

The easy solution is to get rid of the pyproject.toml file. But it would actually be good to be able to use it (because of real build dependencies).

More generally, the solution can not be a pip option. There are many situations (for example install with tox) where you cannot use a pip option.

[pradyunsg]

Here's a workaround I suggest to you: install all build-system.requires items that you want available manually (for eg, via pip install setuptools wheel numpy cython jinja2 fluidpythran>=0.1.7) prior to running pip install --no-isolation ....

---

Please include the pip version you are using and other details. They're in the template for good reasons. They're not in this template. Aha!

What version of pip are you using? You are mentioning PEP 517, but PEP 517 isn't out in a public release yet.

i don't think it is a solution !

If you disable isolation, the user has to manually install the build-time dependencies that they want to be available. The docstring is written assuming that all PEP 518 build-system.requires are indeed required, as defined in the PEP. Perhaps, the doc-string can use improvement.

There are many situations (for example install with tox) where you cannot use a pip option.

tox can: https://tox.readthedocs.io/en/latest/example/basic.html#further-customizing-installation

Honestly, if your CLI tool doesn't allow passing custom options to pip, I don't think that would be pip's problem.

[paugier]

I get this (correct) behavior with the last release of pip (18.1). it is why I didn't mention it.

[dstufft]

Adding more general features to pyproject.toml requires a PEP to extend the standard.

[paugier]

Really, this option --no-isolation is not a solution for us.

1. I'd like to use the build-system.requires option of pyproject.toml. It would allow users to just do pip install fluidfft. Of course I don't want to use --no-isolation because with this option, build-system.requires is not taken into account and we loose all the benefice of pyproject.toml.

2. We need to be able to detect in setup.py if some packages are installed and if they are installed to use them during the build.

[paugier]

tox can: https://tox.readthedocs.io/en/latest/example/basic.html#further-customizing-installation

Good to know! But once again, it is not a solution to this problem.

[paugier]

Adding more general features to pyproject.toml requires a PEP to extend the standard.

PEP 518 is still "Provisional". Could it be modified to add a way to declare optional dependencies?

[dstufft]

Not likely, Provisional in this case pretty much just means it's accepted, but it hasn't been implemented/released yet so we might need to make small tweaks still in order to make it functional in the real world.

Adding new features is generally out of scope.

[paugier]

Then packages with optional build dependencies CAN NOT (and won't be able to) use pyproject.toml ?

it is a problem! it is not something completely crazy to have a try: import ... except ImportError: in a setup.py.

Note that the solution with build-system.optional seems very simple to implement.

[dstufft]

I mean, then the question becomes, what if there are two things listed in optional? Do we need to install both or neither? Or can we install just one of them? Should we be trying to opportunistically install them and fail silently if we cant, or should they be opt in or opt out?

What if we want to have groups of optional dependencies? Should that be supported? All of the same questions asked above apply to that too.

None of these questions are, on the surface, super hard. But they all require sitting down and working through the options and making a decision about how they should function. That's something that deserves wider discussion.

By all means, you're free to try and raise a discussion on the discuss.python.org Packaging category or on distutils-sig to see if other people are open to amending PEP 518 for this feature. I just think it's more likely going to be the case that we want an additional PEP for that feature.

[pfmoore]

I agree with @dstufft. There's nothing fundamentally difficult to decide here, but it needs to be driven by use cases and a reasonable sense of what's important and what is generalisation for its own sake.

I see this as being a follow-up PEP that extends PEP 518, and not suitable as an amendment to PEP 518. If you want to propose and champion such a PEP, then as @dstufft says, you need to start the discussion on distutils-sig or the Discource category.

[dstufft]

That being said, PEP 518 is released in pip now I think yea? We should probably move it out of provisional anyways.