Removal of escaping in shebang in pip install causes "failed to create process" on Windows

[ghost]

Originally reported by: joeyespo (Bitbucket: joeyespo, GitHub: joeyespo)

---

I have Python installed in C:\Program Files (x86)\Python (for legacy reasons--it's kind of late to move it now), and any console_script I install from PyPI breaks now. Running the script causes a "failed to create process" error.

I noticed that older console_scripts were working, and turns out it's because they were quoted. Ex: #!"C:\Program Files (x86)\Python\python.exe"

Manually adding quotes around the newly installed projectname-script.py fixes it.

It looks like the culprit may have been the fix in #188. From that issue:

If it is possible to detect being on a posix platform (probably not that easy?), we could just skip the escaping, as it is never beneficial. If this is not possible, maybe just skip the escaping all together? The benefit seems very platform specific.

Turns out skipping escaping all together just threw the problem back over the fence to Windows. Perhaps it'd be best to detect the platform? Escaping is indeed a platform-specific thing.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/398

[ghost]

Original comment by joeyespo (Bitbucket: joeyespo, GitHub: joeyespo):

---

I'm not sure if this is a good idea or not, but one way to fix this without involving escaping would be to use ctypes.windll.kernel32.GetShortPathNameW when it's available. This gives you the "short" filename that will never contain spaces. It might be appropriate if the complexity of escaping is high and if the filename here is isolated (as in not public / not used anywhere else).

Details: http://stackoverflow.com/questions/23598289/how-to-get-windows-short-file-name-in-python

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I'm -1 for using the short name. I'd like to devise a solution that supports all legal file paths with their canonical names.

[ghost]

Original comment by joeyespo (Bitbucket: joeyespo, GitHub: joeyespo):

---

@jaraco Agreed. +1 to that.

[ghost]

Original comment by johnthagen (Bitbucket: johnthagen, GitHub: johnthagen):

---

Just wanted to add that this problem manifests itself in a slightly different way when Python 3.5 is installed by default on Windows. If the user has a space in their name (e.g. John Doe), then this problem arises and is very difficult to debug.

Example path:

C:\Users>where.exe python
C:\Users\John Hagen\AppData\Local\Programs\Python\Python35\python.exe

pypa/pip#2783 (comment)

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Before I switched to OS X in May, I ran Windows almost exclusively, but I didn't encounter this issue because I used SETUPTOOLS_LAUNCHER=natural.

Still, I would not expect this issue to arise even without that setting as the script generation is specifically special-cased for Windows.

I'll take a quick look to see if I can replicate the issue.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I'm not able to replicate the issue simply. Here's my install of setuptools on Python 3.5 in a directory with spaces and evidence that the script shebang includes quotes:

C:\Users\jaraco> & 'C:\Program Files\Python 3.5\python.exe' -m easy_install setuptools
Searching for setuptools
Best match: setuptools 18.5
Adding setuptools 18.5 to easy-install.pth file
Installing easy_install-script.py script to C:\Program Files\Python 3.5\Scripts
Installing easy_install.exe script to C:\Program Files\Python 3.5\Scripts
Installing easy_install-3.5-script.py script to C:\Program Files\Python 3.5\Scripts
Installing easy_install-3.5.exe script to C:\Program Files\Python 3.5\Scripts

Using c:\python\lib\site-packages
Processing dependencies for setuptools
Finished processing dependencies for setuptools
C:\Users\jaraco> cat C:\python\scripts\easy_install-3.5-script.py
#!"C:\Program Files\Python 3.5\python.exe"
# EASY-INSTALL-ENTRY-SCRIPT: 'setuptools==18.5','console_scripts','easy_install-3.5'
__requires__ = 'setuptools==18.5'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('setuptools==18.5', 'console_scripts', 'easy_install-3.5')()
    )
C:\Users\jaraco> echo $env:SETUPTOOLS_LAUNCHER
executable

Can someone whose environment has this issue what shebang appears when installing setuptools itself? If it doesn't include the quotes, can you trace the code to determine what in your environment might differ from mine? Or alternately, how can I configure my environment to replicate the behavior you see?

[ghost]

Original comment by johnthagen (Bitbucket: johnthagen, GitHub: johnthagen):

---

@jaraco I have been able to reproduce the error with the following steps on two Windows 10 x64 machines and a Windows 10 x64 VM.

1. Install Windows 10 x64 (though I suspect that this problem exists on at least Windows 8 and 7 as well).
2. Create a user with a space in their name (e.g. John Doe)
3. Log into John Doe.
4. Install Python 3.5.1 (64-bit) and select the option to add python to PATH.
5. > pip install cpplint
6. > cpplint

failed to create process.

Note that what is actually failing here is the console_script .exe file (cpplint.exe) that pip creates when it installs the package.

If you easy_install cpplint instead of using pip, it works. So it seems to be a bug in how pip creates the .exe files, related to having a space in some path.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Aha. In that case, the issue lies with pip and distlib. I'm not sure whether to file the issue with the former or the latter, but I'd start with the former, as you're using pip, so the issue applies to it, and if an upstream bug needs to be filed with distlib, the pip team can advise.

[tomduck]

See pip Issue #2783 for a discussion of the same problem, including a workaround that can be used in setup.py.

[fkrull]

I did some digging into this issue; it's either a setuptools or distutils problem. I'll stick to cpplint, but any package using setuptools and entry points should show this behaviour. Things I have observed:

1. pip install cpplint installs the wheel and works fine.
2. pip install --no-binary cpplint cpplint doesn't quote the shebang properly. It installs from sdist by running setup.py install --single-version-externally-managed locally.
3. setup.py install from the source installs fine.
4. setup.py install --single-version-externally-managed from the source doesn't quote the shebang line properly.

I don't know what the egg installation code does different, but in setuptools/command/install_scripts.py line 33 exec_param is retrieved from the build_scripts command which in turn gets it from the build command (distutils/command/build_scripts.py line 38) which, if unspecified, sets it to this (`distutils/command/build.py line 120):

self.executable = os.path.normpath(sys.executable)

Then, exec_param -- which is just an unquoted path, possibly containing spaces -- gets passed to CommandSpec.from_param (setuptools/command/install_scripts.py line 42) which parses it as a command line by splitting it along spaces (setuptools/command/easy_install.py line 1954). As a consequence, the special quoting code gets bypassed.

Wrapping exec_param in a one-element list solved the problem (not quite a one-line fix because it shouldn't be wrapped if it's None). I can roll a PR, but I don't know the setuptools codebase well enough to determine if this is the best place to fix this.

[anthrotype]

You're right, it's a setuptools issue, not pip or distlib.

When installing from a binary wheel (or when installing from sdist package), pip uses distlib to create the console/gui scripts, and distlib does take care of "enquoting" the executable in the shebang:

https://github.com/pypa/pip/blob/master/pip/_vendor/distlib/scripts.py#L161-L164
https://github.com/pypa/pip/blob/master/pip/_vendor/distlib/scripts.py#L66-L79

I think it's in easy_install.ScriptWriter.get_args where the executable enquoting should go.

[anthrotype]

I confirm what @fkrull just said.

The executable string from distutils build_scripts command is being split using shlex inside easy_install's CommandSpec.from_string.

Passing [exec_param] as a one-element list (when not None) solves the issue, as CommandSpec._render method will then concatenate the strings with subprocess.list2cmdline, which in turn correctly escapes spaces with double quotes following MS rules.

Thanks for finding this out! 👍