pyproject.toml prevents `pip install --user --editable` (v19.0.3)

[con-f-use]

Environment

• pip version: pip 19.0.3
• Python version: Python 3.6.7
• OS: Ubuntu bionic 18.04.2 LTS

Description
There seems to be no way to install an editable package in userspace with a pyproject.toml file present. At least on Ubuntu, Debian, possibly other systems while not using a Python virtual environment.

Given the following minimal project, both pip install --user -e . and pip install -e . fail if, and only if the pyproject.toml file is present:

$ tree
.
|____setup.py
|____pyproject.toml
|____src
| |______init__.py

$ find . -type f -exec echo -e '-----\n#{}' \; -exec cat {} \;
-----
#./setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(
    name='nouser',
    version='0.1',
    packages=find_packages('src'),
    package_dir={'src': 'src'},
)
-----
#./pyproject.toml
[build-system]
requires = ['setuptools', 'wheel']
-----
#./src/__init__.py
def oh_hell():
    print('hello')
oh_hell()

Expected behavior

With a pyproject.toml present next to setup.py, either

pip install -e .

or

pip install --user -e .

installs the package in userspace without special privileges and python -c 'import nouser; nouser.oh_hell()' prints hello successfully.

Actual behavior

$ ls
pyproject.toml setup.py  src
$ pip install -e .
Looking in indexes: https://pypi.org/simple/
Obtaining file:///tmp/tmp.I6P01cKtsE
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: nouser
  Running setup.py develop for nouser
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps:
    running develop
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/test-easy-install-15379.write-test'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /usr/local/lib/python3.6/dist-packages/
    
    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.
    
    For information on other options, you may wish to consult the
    documentation at:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html
    
    Please make the appropriate changes for your system and try again.
    
    
    ----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /tmp/tmp.I6P01cKtsE/

$ ls
pyproject.toml setup.py  src
$ pip install --user -e .
Looking in indexes: https://pypi.org/simple/
Obtaining file:///tmp/tmp.I6P01cKtsE
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: nouser
  Running setup.py develop for nouser
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized
    
    ----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /tmp/tmp.I6P01cKtsE/

But without pyproject.toml:

$ ls
pyproject.toml setup.py  src
$ rm pyproject.toml 
$ ls
setup.py  src
$ pip install --user -e .
Looking in indexes: http://10.17.65.203/root/devel/+simple/, https://pypi.org/simple/
Obtaining file:///tmp/tmp.I6P01cKtsE
Installing collected packages: nouser
  Running setup.py develop for nouser
Successfully installed nouser

Additional Information

Running the listed command manually installs the package without problems:

/usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/tmp.I6P01cKtsE/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=
running develop
running egg_info
writing src/nouser.egg-info/PKG-INFO
writing dependency_links to src/nouser.egg-info/dependency_links.txt
writing top-level names to src/nouser.egg-info/top_level.txt
reading manifest file 'src/nouser.egg-info/SOURCES.txt'
writing manifest file 'src/nouser.egg-info/SOURCES.txt'
running build_ext
Creating /home/jan/.local/lib/python3.6/site-packages/nouser.egg-link (link to src)
Adding nouser 0.1 to easy-install.pth file

Installed /tmp/tmp.I6P01cKtsE/src

[cjerdonek]

Fixing this issue is the purpose of this PR: #6370

The one difference from what you described in your "expected behavior" section though is that you'll also need to pass --no-use-pep517. (Also, the contents of pyproject.toml can't be such that PEP 517 mandates processing the source tree as pyproject.toml-style.)

There's a pretty detailed description as the first comment in the PR. It would be good to hear your feedback.

[cjerdonek]

See also issue #6314 for additional discussion.

[con-f-use]

Thanks for the cross-references and your efforts. I'm not even sure what the behavior is supposed be. I'm especially vague on how setuptools is supposed to interact with pip and the pyproject.toml file. There is sadly very little information attainable for a "casual developer".

I have many questions, but to keep matters concrete, am I correct in these points?

• In my case, where I have nothing but a requires = ['setuptools', 'wheel'] the behavior should be to install the required packages, then fall back to calling setup.py
• Pip falls back to simply calling setup.py inside a new virtual environment purely for installation/package building with some tricks to make the venv part work
• Somewhere the trickery is too elaborate, and that is why installation fails
• Because --no-use-pep517 should cause pyprojefct.toml to be ignored, the installation should succeed when using --no-use-pep517 (it does not).
• Invocations with --use-pep517 are supposed to be the cases, where the problem should most certainly manifest
• PEP517 is provisionally accepted and should be the standard (as long as there is a pyproject.toml). Except for the failing installation part, all of the above is the expected behavior. If installation had succeeded, no one would scratch their heads, --use-pep517 would be superfluous in the case of my first post example because it's the default when nothing else is specified.

[cjerdonek]

Your summary seems mostly or roughly correct at points, but I wouldn't want to say it's correct without qualification for fear of leading you down the wrong path. For example, here are a couple clarifications (there could be others):

Somewhere the trickery is too elaborate, and that is why installation fails

It's failing not because it's too elaborate, but because it wasn't meant to support this case. It should be failing, but for a different reason. More explanation is below.

Because --no-use-pep517 should cause pyprojefct.toml to be ignored, the installation should succeed when using --no-use-pep517

It's not that pyproject.toml should be ignored. Rather, --no-use-pep517 should cause the PEP 517 process not to be used in your particular case. (The pyproject.toml file should still be read for the build dependency info.) Again, more below.

To restate some things, PEP 517 introduced a new way of building packages that involves setting up an isolated environment. I would encourage you to take a look: https://www.python.org/dev/peps/pep-0517/

One important aspect missing from your points is any mention of editable mode. It's important because that's what your issue involves.

Since PEP 517 doesn't support editable mode, pip should be erroring out / failing fast if the user tries to use the new PEP 517 style with editable mode (but it's not failing fast). Currently, I would say whatever it's doing now is unsupported / undefined, which I believe is the behavior you're running into.

Feel free to ask specific questions for further clarification.

[con-f-use]

Okay thanks again, I appreciate you taking the time. You've answered everything relevant for my issue.

Just to summarize: The build dependencies in pyproject.toml should be installed when calling pip install --no-use-pep517 --user --editable ., but then pip should just go on without the pep517 process (most notably not creating a virtual environment for building the package). Correct?

If so, it doesn't seem to be:

$ cat pyproject.toml 
[build-system]
requires = ['setuptools', 'wheel', 'gitpython']

$ cat setup.py 
#!/usr/bin/env python
import git
from setuptools import setup, find_packages
setup(
    name='nouser',
    version='0.1',
    packages=find_packages('src'),
    package_dir={'': 'src'},
)

$ pip install --no-use-pep517 --verbose --user -e .
Created temporary directory: /tmp/pip-ephem-wheel-cache-ri4yj0ls
Created temporary directory: /tmp/pip-req-tracker-2zhosv5x
Created requirements tracker '/tmp/pip-req-tracker-2zhosv5x'
Created temporary directory: /tmp/pip-install-hvvea28r
Obtaining file:///tmp/tmp.M7szgoNiRs
  Added file:///tmp/tmp.M7szgoNiRs to build tracker '/tmp/pip-req-tracker-2zhosv5x'
    Running setup.py (path:/tmp/tmp.M7szgoNiRs/setup.py) egg_info for package from file:///tmp/tmp.M7szgoNiRs
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/tmp.M7szgoNiRs/setup.py", line 2, in <module>
        import git
    ModuleNotFoundError: No module named 'git'
Cleaning up...
Removed file:///tmp/tmp.M7szgoNiRs from build tracker '/tmp/pip-req-tracker-2zhosv5x'
Removed build tracker '/tmp/pip-req-tracker-2zhosv5x'
Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp.M7szgoNiRs/
Exception information:
Traceback (most recent call last):
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 158, in prep_for_dist
    self.req.prepare_metadata()
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 530, in prepare_metadata
    self.run_egg_info()
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 609, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/jan/.local/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 761, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmp.M7szgoNiRs/

---

Interlude

To elaborate on another question I had:

• Why is editable mode not regulated by PEP517/518?

The answer seems to be, because it's complicated:

This PEP [517] originally specified another hook, install_editable, to do an editable install (as with pip install -e). It was removed due to the complexity of the topic, but may be specified in a later PEP.

Briefly, the questions to be answered include: what reasonable ways existing of implementing an 'editable install'? Should the backend or the frontend pick how to make an editable install? And if the frontend does, what does it need from the backend to do so.

[cjerdonek]

If so, it doesn't seem to be:

Yeah. That's (one of) the issues that PR #6370 is meant to fix. It would be great if you could try installing from that branch and trying the fix, if you're able.

Why is editable mode not regulated by PEP517/518? The answer seems to be, because it's complicated:

Well, not necessarily that it's complicated per se, but more that the PEP process itself was complicated. The PEP 517 discussion was already very long (several months) and contentious and difficult to get agreement on other aspects, so discussion of editable mode was removed in the interests of simplifying things and getting agreement on the other aspects.

[cjerdonek]

This has now been addressed by PR #6370. (Note that the issue number of this issue wasn't referenced by PR #6370 since that PR was created before this issue was filed.)

@con-f-use If there are any PEP 517 / editable mode issues you think still remain to be addressed, feel free to open a new issue specific to that concern.