some pip editable installs don't work with PEP 518 build isolation

[benoit-pierre]

> mkdir /tmp/simple_pep518_project-1.0
> printf 'from setuptools import setup; setup(name="simple_pep518_project", version="1.0")\n' >/tmp/simple_pep518_project-1.0/setup.py
> printf '[build-system]\nrequires = ["setuptools", "wheel"]\n' >/tmp/simple_pep518_project-1.0/pyproject.toml
> pip install --prefix /tmp/pyprefix -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix:
    running develop
    Checking .pth file support in /tmp/pyprefix/lib/python3.6/site-packages
    /usr/bin/python -E -c pass
    TEST FAILED: /tmp/pyprefix/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH
    
    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /tmp/pyprefix/lib/python3.6/site-packages
    
    and your PYTHONPATH environment variable currently contains:
    
        '/tmp/pip-build-env-q4e8bpkm/lib/python3.6/site-packages'
    
    Here are some of your options for correcting the problem:
    
    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files
    
    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)
    
    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
    
    
    Please make the appropriate changes for your system and try again.
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix" failed with error code 1 in /tmp/simple_pep518_project-1.0/
> pip install --user -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /tmp/simple_pep518_project-1.0/

This is because of pip's build isolation when PEP 518 support is enabled and easy_install's behavior of:

• conflating the python environment used during installation with the target environment, erroneously assuming that the package target directory must be in PYTHONPATH.
• disabling support for the --user option if the user site is disabled:

> python easy_install.py -h | grep -- --user                                                                  
  --user                     install in user site-package
> python -s easy_install.py -h | grep -- --user                                                               
> python -s easy_install.py --user setuptools                                                                 
usage: easy_install.py [options] requirement_or_url ...
   or: easy_install.py --help

error: option --user not recognized

With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in PYTHONPATH, I think:

• the --user option should always be available (with maybe a warning if the user site is disabled)
• the error above when installing to a directory not in PYTHONPATH should be turned to a warning

For reference, related pip issue: pypa/pip#5317

[yan12125]

I got a similar error message. Does that mean pip install --user -e . won't work for projects with pyproject.toml?

[benoit-pierre]

You mean pip install --user -e .?

[yan12125]

Oh yes, sorry for the typo. Benoit Pierre <notifications@github.com> 於 2018年8月2日 週四 01:33 寫道：

…

You mean pip install --user -e .? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1405 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2RGTD8cVKieHbdjPMNpCDQcfyCDK5Pks5uMeZogaJpZM4U7Sol> .

[benoit-pierre]

Unfortunately, yes: you'll have to manually install the required build dependencies and use --no-build-isolation to work around the issue.

[yan12125]

Thanks for the workaround. That works!

[revmischa]

This is a problem affecting me big time. I'm trying to install modules with pip into a target for packaging up for AWS lambdas. This breaks a lot of automation (for https://github.com/awslabs/aws-sam-cli)