x.py: Support systems with only python3 not python

[dtolnay]

Fixes #71818 without the pitfalls so far described in previous attempts.

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[Mark-Simulacrum]

I've never done much development on Windows, but is bash typically available there?

FWIW, I believe the current trajectory is to add an x.ps1 PowerShell script (not sure on extension, I think that's right?) which would be ~equivalent to the x.py wrapper, so even if python isn't available adding that might let us move forward regardless.

[est31]

cc also python/cpython#90803

[ChrisDenton]

I've never done much development on Windows, but is bash typically available there?

msys2 and git bash are development platforms that emulates a nix environment on Windows, including common gnu tools and bash, In those environments this file will be run in bash.

"Native" Windows shells will simply treat this as a python file. So it will run whatever python version is associated with .py files. No bash involved.

[joshtriplett]

Does this need to use bash rather than sh?

[jyn514]

I tried this in #80585 and it broke because powershell didn't know how to run it. Can you confirm this works on windows with powershell?

I had other issues there, but I think they might have been because I intentionally omitted the shebang. IIRC the reason for omitting the shebang is because there's something called py on Windows that special cases the /usr/bin/python and /usr/bin/python3 shebangs, and will give an error for anything else. Can you confirm this works when using py x.py, or if py is the default interpreter for python files?

[ChrisDenton]

because there's something called py

Oh hm. Right the py launcher again.

If python is installed via the Microsoft store, winget, manually, etc then you don't get this "helpful" utility and everything works fine because Windows shells don't read shebangs.

If python is installed via the official installer from python.org then that will install the py launcher which does read shebangs (and therefore fails). The user could manually fix this by changing the file association (either to the real python or to run py -3). But if user invention is required anyway then that defeats the point of finding python automatically.

[jyn514]

@ChrisDenton to be clear, the issue isn't 2 vs 3 - x.py already handles trying to relaunch itself with 3 if necessary. The problem I hit was that py only understands the /usr/bin/python shebang, because it doesn't run in an msys environment and thinks /usr doesn't exist if you try /usr/bin/env bash.

[ChrisDenton]

@jyn514 Yes. Sorry I wasn't clear there. What I'm saying is that the installer sets .py files to be run by the py.exe launcher. This launcher, as you say, reads the file shebang. However, if the user changes the .py association to be run by py.exe -3 then py.exe skips reading the shebang. They could also use py.exe -2 (if they also have python 2 installed). This will also skip reading the shebang. (Edit more succinctly: using py -3 or py -2 instead of plain py skips reading the shebang)

Or they could set the association to run python.exe directly and skip the middleman.

[dtolnay]

Can you confirm this works on windows with powershell?

Not a proficient Windows user, but toying around in a VM, I was not able to get ./x.py to run in powershell neither before nor after this change, neither with Python from Microsoft Store nor python.org. Here is what I found:

setup	./x.py works before?	./x.py works after?	workaround
Linux distro pre-2020	yes	yes
Ubuntu 20.04+, other recent Linux	no	yes	install python-is-python3 or similar, or python3 x.py
macOS pre-2022	yes	yes
macOS Monterey+	no	yes	python3 x.py (is there a thing to install?)
Windows, MSYS2 or git bash, any Python	yes	yes
Windows Subsystem for Linux	yes	yes
Windows, Powershell, Python from MS	no	no	set file association to python.exe, or python3 x.py (py -3 does not work)
Windows, Powershell, python.org	no	no	set file association to py.exe -3, or py -3 x.py (python3 does not work, it opens Microsoft Store)

[dtolnay]

I think my powershell result agrees with #71818 (comment): the shebang currently on master already doesn't work:

so before this PR we manage to be broken on all modern environments across Linux, macOS, and Windows. 😅

[jyn514]

@wesleywiser @rylev @ChrisDenton you use Windows and PowerShell regularly, right? Do you normally use ./x.py? Or python3 x.py?

[dtolnay]

I think @ChrisDenton answered that in #98474 (comment), and I found the same: ./x.py works in Powershell (before and after PR identically) as long as you do the one-time setup of associating to run it with py.exe -3.

However, if the user changes the .py association to be run by py.exe -3 then py.exe skips reading the shebang.

In the "Administrator Command Prompt": ftype Python.File="C:\Windows\py.exe" -3 "%L" %*

[ChrisDenton]

@wesleywiser @rylev @ChrisDenton you use Windows and PowerShell regularly, right? Do you normally use ./x.py? Or python3 x.py?

The x.exe usually. But ./x or ./x.py also works for me.

[ChrisDenton]

Btw, here's a quick powershell log after installing from python.org in a fresh sandbox:

PS C:\Users\WDAGUtilityAccount> $env:PATHEXT+=';.PY'
PS C:\Users\WDAGUtilityAccount> cat x.py
#!/usr/bin/env bash

'''':
echo "hello from bash"
exit 0
'''

print("hello from powershell")
PS C:\Users\WDAGUtilityAccount> ./x
Unable to create process using '/usr/bin/env bash "C:\Users\WDAGUtilityAccount\x.py" '
PS C:\Users\WDAGUtilityAccount> py x.py
Unable to create process using '/usr/bin/env bash x.py'
PS C:\Users\WDAGUtilityAccount> py -3 x.py
hello from powershell

On the machine I use for development (using the same x.py as above):

PS C:\Users\Chris> ./x
hello from powershell
PS C:\Users\Chris> ./x.py
hello from powershell
PS C:\Users\Chris> python x.py
hello from powershell

It's possible I did some additional setup though. I'm using the python from the Windows store.

[Mark-Simulacrum]

I think @ChrisDenton answered that in #98474 (comment), and I found the same: ./x.py works in Powershell (before and after PR identically) as long as you do the one-time setup of associating to run it with py.exe -3.

However, if the user changes the .py association to be run by py.exe -3 then py.exe skips reading the shebang.

In the "Administrator Command Prompt": ftype Python.File="C:\Windows\py.exe" -3 "%L" %*

I think it'd be great to get this line noted somewhere in our documentation -- maybe the rustc-dev-guide at least? We could consider emitting a one-line command if it's not the case (presumably detectable?) from x.py or similar.

In the meantime I'm happy with the table (#98474 (comment)) and other comments here sufficiently to let us move forward; we can revisit if this causes regressions in practice. Thanks @dtolnay for pushing this forward!

@bors r+ rollup

[bors]

📌 Commit 9169905 has been approved by Mark-Simulacrum

[dtolnay]

I can follow up with a docs PR. I know almost nothing about Windows beyond what's discussed above and in the issue, so maybe it would be even better if someone regularly doing development on Windows could handle the docs.

Here is a draft:

---

PowerShell users: if ./x.py gives you an error saying "Unable to create process using '/usr/bin/env bash …'" then:

• If py -3 x.py seems to run successfully, you can run ftype Python.File="C:\Windows\py.exe" -3 "%L" %* in the "Administrator Command Prompt" to make x.py by itself work.

• If py -3 x.py does not work but python3 x.py does, you can run ???? to make x.py by itself work.

---

Maybe we could put this not just in the rustc-dev-guide but also directly into x.py, since that's probably where I would first look after a failure.

[wesleywiser]

@wesleywiser @rylev @ChrisDenton you use Windows and PowerShell regularly, right? Do you normally use ./x.py? Or python3 x.py?

@jyn514 I use python x.py from Powershell.

[CAD97]

This broke x.py on Windows machines. I've been using py ./x.py without trouble until my most recent git pull. I typically don't use x.py directly, but it also functioned.

Specifically, running py ./x.py from nushell or PowerShell now tries and fails to launch /usr/bin/env bash.

(Windows 11)

I'm also fairly certain I did install python via winget, though I don't know which Winget package I used (Winget doesn't seem to be telling).

[CAD97]

In general, it seems suspicious to have a .py with a bash shebang. imho this should've been a x.sh.

[Mark-Simulacrum]

I'd prefer to avoid a revert without more information - at the very least, to determine what we can do to test in the future, given that comments on this thread suggest to me that the file should continue to work with py x.py...

[jyn514]

@Mark-Simulacrum I think @dtolnay tested with ./x.py, but not py x.py.

[Mark-Simulacrum]

Ah, I see. #98474 (comment) has the bash error that's presumably not working - py x.py doesn't work, but apparently py -3 x.py does, and presumably ftype Python.File="C:\Windows\py.exe" -3 "%L" %* then works.

[CAD97]

I can confirm that the status on my install is (from cmd)

D:\git\rust-lang\rust>winget list python
Name            Id                                     Version     Available   Source
-------------------------------------------------------------------------------------
Python 3        Python.Python.3                        3.10.4150.0 3.10.5150.0 winget
Python Launcher {691AAAA1-FE86-4973-8DA2-6AA2B3327562} 3.10.7751.0

D:\git\rust-lang\rust>ftype | rg py
Python.ArchiveFile="C:\Windows\py.exe" "%L" %*
Python.CompiledFile="C:\Windows\py.exe" "%L" %*
Python.File="C:\Windows\py.exe" "%L" %*
Python.NoConArchiveFile="C:\Windows\pyw.exe" "%L" %*
Python.NoConFile="C:\Windows\pyw.exe" "%L" %*

D:\git\rust-lang\rust>py ./x.py --help
Unable to create process using '/usr/bin/env bash ./x.py --help'

D:\git\rust-lang\rust>x.py --help
Unable to create process using '/usr/bin/env bash "D:\git\rust-lang\rust\x.py"  --help'

D:\git\rust-lang\rust>py -3 ./x.py --help
info: Downloading and building bootstrap before processing --help
      command. See src/bootstrap/README.md for help with common
      commands.
Building rustbuild
    Finished dev [unoptimized] target(s) in 0.06s
Usage: x.py <subcommand> [options] [<paths>...]

I did install python through winget, but I installed Python.Python.3, not the msstore python. I also manually uninstalled the msstore python3 shim.

Note that on Windows, the story about when you have to use ./x.py or x.py is... inconsistent. cmd requires x.py; ./x.py will fail to find a program .. pwsh requires ./x.py, and x.py will not work unless you add . to $env:PATH, to avoid pwd files overriding programs loaded from PATH.

Because of this, even if I change my .py file handler, my muscle memory is still to call py ./x.py, because this works both in pwsh and cmd (and nushell, what I'm actually daily driving atm, uses cmd to run externals, so behaves like cmd).

---

The proper fix is for py.exe to stop trying (and failing horribly) to interpret shebangs that aren't python shebangs. If you're running py, you intend to run python.

[CAD97]

I opened an issue: #98650

[jyn514]

I think we should revert this change.

Before this PR

Unix

You need python in PATH. The errors when using ./x.py on Linux are fairly reasonable ("command python not found"). People sometimes run into issues when they first start contributing, but they're easy to work around.

Windows

• ./x.py doesn't work because there's no program associated with it
• py, python3, python2 all work.

After this PR

Unix

You need bash in path, and any of python3, python, or python2 installed.

Windows

• py breaks with unhelpful errors ("/usr/bin/env not found"). This breaks existing workflows, not new contributors.
• ./x.py still doesn't work.

[CAD97]

That's not quite a correct summary of the Windows behavior.

Before (#!/usr/bin/env python)

• ./x.py (pwsh; x.py in cmd) works iff an ftype handler has been registered for .py; this is the case if the “full installer” was used, or if the msstore python was used and a default “Open With” has been manually set to a python interpreter.
• With a “full installer” python, py ./x.py and python ./x.py work; python2 ./x.py and python3 ./x.py do not.
• With msstore python, python2 or python3 (as applicable) may work; I tested installing msstore python and did not get a python3 exe in path, but this behavior may be different if the default python3 shim which launches msstore has not been uninstalled.

After (#!/usr/bin/env bash)

• py ./x.py (and ./x.py if py is the ftype registration) attempt to emulate the behavior of env bash, failing with an unhelpful error if bash is not in path, trying to run /usr/bin/env. (If a bash is available, it is run.)
• python ./x.py, python3, and python2 work as before.

Which shebangs work on Windows?

Through py.exe, any /usr/bin/env python{version} is documented as launching via py -{version}; this applies with any vcommand shebang. Through an MSYS like shell, the shebang is always interpreted and the env program is always run like on any other POSIX shell.

As such, #! /usr/bin/env python is the most Windows-portable shebang line. python3 is probably also completely acceptable; imho it is not unreasonable to expect a POSIXy shell environment on Windows to provide python3 as a (soft) alias to (the logic of) py -3.

#! /usr/bin/env x where env x fails lookup may start to call the default python when launched via py.exe with Python 3.11+, and may even skip path lookup entirely. It's impossible to say for sure until it releases whether the current implemented behavior diverging from the existing launcher is desired or will be changed to match it. However, trying to envoke /usr/bin/env does seem to be considered a bug.

[ChrisDenton]

For completeness I'd add that installing the py.exe launcher is optional (though default) in the python.org installer.