some pip editable installs don't work with PEP 518 build isolation

[benoit-pierre]

> mkdir /tmp/simple_pep518_project-1.0
> printf 'from setuptools import setup; setup(name="simple_pep518_project", version="1.0")\n' >/tmp/simple_pep518_project-1.0/setup.py
> printf '[build-system]\nrequires = ["setuptools", "wheel"]\n' >/tmp/simple_pep518_project-1.0/pyproject.toml
> pip install --prefix /tmp/pyprefix -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix:
    running develop
    Checking .pth file support in /tmp/pyprefix/lib/python3.6/site-packages
    /usr/bin/python -E -c pass
    TEST FAILED: /tmp/pyprefix/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH
    
    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /tmp/pyprefix/lib/python3.6/site-packages
    
    and your PYTHONPATH environment variable currently contains:
    
        '/tmp/pip-build-env-q4e8bpkm/lib/python3.6/site-packages'
    
    Here are some of your options for correcting the problem:
    
    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files
    
    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)
    
    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
    
    
    Please make the appropriate changes for your system and try again.
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/tmp/pyprefix" failed with error code 1 in /tmp/simple_pep518_project-1.0/
> pip install --user -e /tmp/simple_pep518_project-1.0
Obtaining file:///tmp/simple_pep518_project-1.0
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Installing collected packages: simple-pep518-project
  Running setup.py develop for simple-pep518-project
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --user not recognized
    
    ----------------------------------------
Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/simple_pep518_project-1.0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --user --prefix=" failed with error code 1 in /tmp/simple_pep518_project-1.0/

This is because of pip's build isolation when PEP 518 support is enabled and easy_install's behavior of:

• conflating the python environment used during installation with the target environment, erroneously assuming that the package target directory must be in PYTHONPATH.
• disabling support for the --user option if the user site is disabled:

> python easy_install.py -h | grep -- --user                                                                  
  --user                     install in user site-package
> python -s easy_install.py -h | grep -- --user                                                               
> python -s easy_install.py --user setuptools                                                                 
usage: easy_install.py [options] requirement_or_url ...
   or: easy_install.py --help

error: option --user not recognized

With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in PYTHONPATH, I think:

• the --user option should always be available (with maybe a warning if the user site is disabled)
• the error above when installing to a directory not in PYTHONPATH should be turned to a warning

For reference, related pip issue: pypa/pip#5317

[yan12125]

I got a similar error message. Does that mean pip install --user -e . won't work for projects with pyproject.toml?

[benoit-pierre]

You mean pip install --user -e .?

[yan12125]

Oh yes, sorry for the typo. Benoit Pierre <notifications@github.com> 於 2018年8月2日 週四 01:33 寫道：

…

You mean pip install --user -e .? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1405 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2RGTD8cVKieHbdjPMNpCDQcfyCDK5Pks5uMeZogaJpZM4U7Sol> .

[benoit-pierre]

Unfortunately, yes: you'll have to manually install the required build dependencies and use --no-build-isolation to work around the issue.

[yan12125]

Thanks for the workaround. That works!

[revmischa]

This is a problem affecting me big time. I'm trying to install modules with pip into a target for packaging up for AWS lambdas. This breaks a lot of automation (for https://github.com/awslabs/aws-sam-cli)

[gaborbernat]

@pganssle can you confirm a PR doing:

With easy_install being deprecated, and pip being perfectly happy to install to a target directory not in PYTHONPATH, I think:
the --user option should always be available (with maybe a warning if the user site is disabled)
the error above when installing to a directory not in PYTHONPATH should be turned to a warning

would be accepted?

[pganssle]

@gaborbernat Better to ask @benoit-pierre, he's also a maintainer of setuptools, and he's the one who opened the issue. 😉

I don't have any objection to it, but I also don't understand what consequences there might be. @benoit-pierre were you uncertain about anything in this?

[benoit-pierre]

No, I think it's the right move.

[ssbarnea]

Can we make this happen?

[ssbarnea]

Was anyone able to find a workaround for this? I often test PRs for python modules and I am hitting this more and more often (as people start adopting PEP 518).

This forces developer to install packages as root if they do not want to user a virtualenv. And not using a virtualenv is exactly what you want for testing compatibility with system python distribution.

[joreiff]

Well, if my PR were to be accepted, we wouldn't need a workaround anymore. 😇
On a serious note, I would be very interested in feedback, since this is an issue that is affecting me currently.