$PIPX_HOME on macOS since release 1.3.0 breaks many scripts

[hhp21]

It appears that since release 1.3.0, when installing pipx on macOS Ventura, it chooses to set $PIPX_HOME to ~/Library/Application Support/pipx, which creates a problem for scripts like awscli where the directory forms part of the shebang, because shebangs are not allowed to have spaces in it.

As recent as release 1.2.1, $PIPX_HOME on macOS Ventura is set to be ~/.local/pipx which doesn't cause problems.

How to reproduce

I was using Python 3.8 installed using pyenv.

pip install pipx==1.3.0
pipx environment 

It prints the below:

Expected behavior

$PIPX_HOME should be /Users/test3/.local/pipx

[Gitznik]

Since 1.3, pipx bases the $PIPX_HOME on platformdirs, see the docs. This package specifies for mac, that the canonical application data dir is ~/Library/Application Support/{app_name}, which is based on the apple docs. /Users/{user}/.local/{app_name} is the canonical path on Linux.

I personally think it makes sense to stick to the platform specific convention.

For an immediate fix to your problem you could symlink the binary you want to use in a shebang to a directory without spaces in the path. You can then safely use that in the shebang.

We should definitely add this as a warning on the docs for mac though IMO.

[hhp21]

Thanks for the explanation. I've updated the title to be more precise.

[Gitznik]

I found quite a lengthy discussion (tox-dev/platformdirs#4) on the platformdirs repo about suporting XDG, which is essentially what you're asking for.

They ended up not supporting it for macOS. There was an interesting comment though on symlinking ~/Library/Application Support to ~/Library/ApplicationSupport. This apparently is done already if you're a Iterm2 user, but IMO this is generally something we should suggest in our docs, when using pipx to install applications that should be used in a shebang.

@gaborbernat as maintainer of platformdirs, do you have any other insight here?

[gaborbernat]

Nothing besides what was already discussed in that issue 🤔

[Gitznik]

I would suggest adding a note to the mac installation docs, in regards to symlinking ~/Library/Application Support to ~/Library/ApplicationSupport then.

[gaborbernat]

Well this only happens for awscli, no?

[Gitznik]

Any application one would use in a shebang I guess. Not that I could think of another example to be fair.

[gaborbernat]

You are welcome to put in a PR to be added to some FAQ section.

[jaraco]

Based on this comment, I propose this issue should be re-opened.

[jaraco]

Well this only happens for awscli, no?

It seems to happen for lots of projects:

• pass2csv
• ranger
• docutils
• mercurial

I suspect it affects any project not using "console scripts" for their console scripts, but instead providing explicit scripts.

I think maybe the reason it doesn't affect console scripts is because pip uses a different executable depending on the type of script installed:

# wrong
 ~ @ head -n 1 $(which pipx)
#!/opt/homebrew/opt/python@3.12/bin/python3.12
 ~ @ head -n 1 $(which hg)
#!/Users/jaraco/Library/Application Support/pipx/venvs/mercurial/bin/python

[chrysle]

It is currently being worked on reverting the platformdirs usage for pipx paths on macOS in #1297, since several issues with it have been reported. On other Unix systems the behaviour will be kept.

Besides, probably this issue should be reported at the platformdirs location, too.

[jaraco]

I see Gitznik in this comment proposed that the issue be solved with in pip or the packages themselves. Personally, I'd prefer to see the issue solved at the core and add support for spaces in shebangs. Let's stop exposing this broken behavior on downstream workarounds.

The symlink workaround is terrible and should be avoided.

[jaraco]

On other Unix systems the behaviour will be kept.

That's too bad, because other Unix users are affected too, except that they avoid having spaces in their directory names and perpetuate the expectation that spaces are invalid.

[jaraco]

I think maybe the reason it doesn't affect console scripts is because pip uses a different executable depending on the type of script installed:

I was mistaken. I thought I'd installed pipx using pipx, but I see on this machine I have pipx installed using homebrew, so it was a bad example. The reason console scripts aren't affected by the shebang in console scripts is because pip provides a workaround for those scripts:

 ~ @ head -n 2 $(which tox)
#!/bin/sh
'''exec' "/Users/jaraco/Library/Application Support/pipx/venvs/tox/bin/python" "$0" "$@"

[Gitznik]

On other Unix systems the behaviour will be kept.

That's too bad, because other Unix users are affected too, except that they avoid having spaces in their directory names and perpetuate the expectation that spaces are invalid.

This is about making sure the default pipx behavior is robust. We can't really prevent users from setting their pipx home to a path containing spaces. I may extend my PR to print a warning if a space is found in the pipx home.

I don't think this project is the right place to discuss whether spaces should or should not be allowed in shebangs.

[jaraco]

Today I learned that the shell I use actually supports spaces in pathnames in shebangs. Since it uses shlex to parse the shebang line, it works if one puts quotes around the interpreter. In xonsh/xonsh#5315, I provide more details and I'm seeking to find places to make a proper solution more widely available.

[frou]

I run pipx environment --value PIPX_BIN_DIR as part of my .bashrc (adding it to PATH), and since updating to pipx 1.5.0, that means there is stderr spam every time I start a shell.

⚠️ Found a space in the home path. We heavily discourage this, due to multiple
    incompatibilities. Please check our docs for more information on this, as well as
    some pointers on how to migrate to a different home path.

[Gitznik]

I run pipx environment --value PIPX_BIN_DIR as part of my .bashrc (adding it to PATH), and since updating to pipx 1.5.0, that means there is stderr spam every time I start a shell.

Yes, this is intended to warn users that having a pipx_home path with spaces in it can lead to a lot of problems. See this discussion for more details.

To solve this you can

1. Move your pipx home to a location without spaces (see our troubleshooting docs) - I would heavily urge you to do this
2. If you're really sure you want to stick to your old path, you can run the command with the -q flag: pipx environment -q --value PIPX_BIN_DIR. This will silence all warnings.

[Gitznik]

I will actually close this issue, as with version 1.5, this is no longer an issue in the default pipx behavior.

[frou]

Thank you. So I just did pipx uninstall-all, then manually deleted the entire ~/Library/Application Support/pipx directory, then pipx install'd the handful of stuff I use pipx for, afresh. And now it's good.

Where I was personally confused (and possibility others who aren't pipx experts will be too) is that I was reading in to this some kind of implication that I, as a user, had made the choice to use ~/Library/Application Support/pipx (which IIRC wasn't the case).

[Gitznik]

Thank you. So I just did pipx uninstall-all, then manually deleted the entire ~/Library/Application Support/pipx directory, then pipx install'd the handful of stuff I use pipx for, afresh. And now it's good.

Where I was personally confused (and possibility others who aren't pipx experts will be too) is that I was reading in to this some kind of implication that I, as a user, had made the choice to use ~/Library/Application Support/pipx (which IIRC wasn't the case).

Gotcha. If you have an idea how to word this more clearly, you're welcome to open a PR :) The warning is generated here:

pipx/src/pipx/paths.py

Lines 114 to 116 in 8fbc085

	f"{hazard} Found a space in the home path. We heavily discourage this, due to "
	"multiple incompatibilities. Please check our docs for more information on this, "
	"as well as some pointers on how to migrate to a different home path."