PEP 394: Allow for more flexibility in handling /usr/bin/python

[hroncok]

Co-Authored-By: @encukou

[ncoghlan]

The proposed change looks good to me in general, some minor wording suggestions and one more substantial adjustment inline.

[encukou]

As co-author, I agree with the change, but I don't think I should be the reviewer :)

[gvanrossum]

You should probably get the Steering Council to appoint a BDFL-Delegate who can approve it. Last time we had MvL as PEP Czar but he's not around, and BDFL-Delegate is the closest thing. I'm not touching this myself. :-)

[encukou]

Thanks for the advice! Just before I read your comment, I asked for, essentially, just that :)
python/steering-council#8

[warsaw]

Since Nick, Petr, and myself are all authors, who would we get to be a BD?

[gvanrossum]

The SC could deliberate it for a bit during our next meeting.

[ncoghlan]

Definitely an SC level discussion (and it probably makes sense for us to just review the PR changes directly).

Some potentially useful context for that discussion:

• https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal (Fedora 30 is feature complete and will be released in the next few weeks)
• https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal (Fedora 31 is scheduled to be released in October)
• https://fedoraproject.org/wiki/Changes/RetirePython2 (Fedora 32 doesn't have a final schedule yet, but late April/early May is the usual time frame - the key point for our purposes is that it will be the first Fedora release after Python 2 goes completely end of life upstream)

[hroncok]

To complete the list of Fedora changes posted by Nick:

My idea is to also include a Fedora 31 change that moves /usr/bin/python (and others, such as nose, pytest, pip) to Python 3 (with possible deferral to F32), together with package provides (dnf install python to install Python 3, dnf install python-requests to install python3-requests). I just haven't drafted it yet, waiting for the resolution of this PEP update.

https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3#Phase_3:_Switch_python_to_refer_to_python3

[ncoghlan]

@hroncok Ah, nice! I think that would tie in better with my proposed guidance then, as once you've implemented that change, then /usr/bin/env python shebangs will start working on Python 3 only Fedora systems, just as they'd work on RHEL 8 systems where an admin has run alternatives --set python /usr/bin/python3 (as per https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/)

[encukou]

About setting /usr/bin/python, that article says:

Note, We do not recommend this approach. We recommend you explicitly refer to python3 or python2. That way, your scripts and commands will work on any machine that has the right version of Python installed.

I still think that is good advice. Setting /usr/bin/python is a decision of the sysadmin, not the user, and it's machine-global – if anything on the machine grows to depend on the decision, it practically can't be changed.
Generic scripts are better off using python3.

Keep in mind this PEP's advice will be applied to new code. New code generally won't be deployed on systems without a python3 command. Not having python3 signalizes an old system with a high cost of upgrading (not only the system, but also software on it).

[ncoghlan]

I don't think always using "/usr/bin/python3" is generically good advice for new code, since I think requiring the python3 spelling is ultimately a transient thing - I expect most distros to follow Fedora's approach, and make python refer to python3 by default once they drop their full Python 2 stacks, and I expect we'll eventually change the Windows launcher to at least fall back to Python 3 for unqualified shebang lines, and potentially even prefer it over Python 2.7 once the latter goes end-of-life.

So I think it's entirely reasonable for publishers to continue to use "/usr/bin/env python" as their default approach, and tell anyone with a system where that doesn't work to either:

1. Use an appropriately configured virtual environment; or
2. Set up a shell alias or user level symlink so it works for their personal account; or
3. Convince the system administrators to make it work by default across the entire machine; or
4. Edit the shebang line to be correct for their system before running the script (if editing a shebang line is problematic, folks should be using installers, not downloading scripts directly)

While specifying /usr/bin/env python3 instead is an entirely reasonable alternative, that offers some potential benefits and is hence well worth mentioning, that's a far cry from telling all current publishers "Your script shebang lines are now wrong, and you should change them" (or, even more irritatingly, encouraging folks to go around submitting PRs to open source projects along the lines of "PEP 394 told me that /usr/bin/env python is wrong, so now I am going around and submitting 'Change python to python3 in shebang lines' PRs to every open source Python project I can find").

[encukou]

I think requiring the python3 spelling is ultimately a transient thing

Is that your personal view, or can the SC agree on that? Last time I suggested this, it was explicitly shot down (#630 (comment)). This PR tries to reflect consensus from the mailing list (as much as possible while giving sound suggestions).

This is really frustrating – it feels like no one has the authoriity to decide, but too many people have a right to veto. And meanwhile, the status quo is inadequate; the PEP needs to be changed now if it's to keep any hope of relevance. This issue is good burnout material.

The explicit python3 will work now. It's good guidance for script authors that want their software to just work on users' machines.

Your script shebang lines are now wrong, and you should change them

Would it be OK to just add a disclaimer that while while the python3 suggestion works, different projects have different needs and other shebangs aren't necessarily wrong?

[ncoghlan]

(I'm currently commenting from a personal point of view, but I expect the merge decision on this PR will be made by SC consensus and/or vote)

The key outcome I want to avoid is anyone claiming "PEP 394 says that using /usr/bin/env python for source compatible scripts is always wrong", so yeah, I'd be OK with wording that made the points:

• /usr/bin/env python3 is compatible with virtual environments, Python 3 only systems, and dual stack systems today, so choose this option if compatibility with newer Python 3 only systems is higher priority than compatibility with older Python 2 only systems (even for source-compatible scripts). This is often going to be the case for open source projects.
• /usr/bin/env python is compatible with virtual environments, Python 2 only systems and dual stack systems today, and appears likely to become increasingly compatible with Python 3 only systems over time (assuming distros continue on the design trajectories that they're currently on), so choose this option if either compatibility with legacy Python 2 only systems is a high priority, or else if you have existing shebang lines that you'd prefer to leave unmodified for now.

[ncoghlan]

This looks good to me now. The big change from the initial PR is to keep the "/usr/bin/env python" recommendation as the default, but emphasise that this may require the use of virtual environments to provide cross platform consistency.

It then frames the additional freedom for Python distributors in terms of making their system Python installations behave more like a Python virtual environment.

[ncoghlan]

@gvanrossum @brettcannon @gvanrossum @willingc I think this is ready for broader steering council review now.

General scope of change:

• we grant Python distributors more freedom to either make the python name refer to the Python 3 runtime (the same way it will in a Python 3 virtual environment), or else make it configurable at the system level (again, more like virtual environments)
• we clarify for publishers some of the trade-offs between the different options for targeting system installations directly

[ncoghlan]

Ping @warsaw. This is just waiting on the question of whether or not you're happy for the stance the PEP takes to be "Unless they specifically know otherwise, script publishers only need to worry about end users that are either going to be using a virtual environment, or else are going to be running a Python package installer".

I think that's the only reasonable stance we can take - at this point, shebang lines are too inconsistent across different platforms for us to do anything else. If you can narrow your focus to a specific subset of target platforms, then there are alternatives available, but that's guidance that documentation for those specific environments should be providing, not our generic upstream suggestions.

[encukou]

ping @warsaw. Care to reply? Anything I can do to make your job easier?

[brettcannon]

I'm going to put this on the steering council meeting agenda until Barry attends and notices this PR. 😄

[encukou]

We have a deadline coming up next Tuesday for Fedora 31 (scheduled to live 2019-10 to 2020-11). I'd like to let users have a working python command there by default.

Since this PR's recommendations for distros are, as far as I know, agreed on, I'm tempted to switch python to python3 even if the PEP is not updated in time.

(edit: corrected Fedora deadline date)

[ncoghlan]

@encukou Aye, that makes sense, as the only thing holding this up are the remaining concerns regarding /usr/bin/env python as a default shebang recommendation for script publishers, even though it's the only venv-compatible option.

We also have a steering council meeting tomorrow morning, so I'll try to get it resolved then - it's the classic problem of "Most software publishers are not system integrators, so what's right for system integrators shouldn't affect the default recommendations" :)

[ncoghlan]

Huh, since I deleted/edited the offending line I can't find a way to get back to the "single comment" thread that @warsaw started on the shebang line recommendation. (Found it via the commit history on the PR)

Anyway, 4b20f8c changes the mention of /usr/bin/env python to be specific to activated virtual environments (based on the 2019-06-26 SC discussion).

[hroncok]

I've read the entire proposal again and from the Fedora POV I'm in favor!

[hroncok]

The GitHub interface had some whitespace problem and I've broke the RST syntax, Sorry about that, should be good now.

[warsaw]

@ncoghlan Thanks very much for the changes regarding /usr/bin/env. With the focus on venvs, I think the recommendations in the PEP are great now. I do have a few suggestions for improvements to the text, but I think these are mostly minor.

[warsaw]

The second sentence feels a little too imprecise. What exactly is an "older Linux distribution" and how many of them count as "most"? Perhaps:

Older Linux distributions will provide a python command that refers to Python 2, and will likely not provide a python2 command

[warsaw]

I'd suggest leaving out any mention of Python 4. 1) it can create FUD; 2) it's entirely vaporware; 3) we don't even know whether there will be a Python 4 or what it will look like. We can always add a Python 4 recommendation when the time comes.

[willingc]

Agreed. Good catch @warsaw

[encukou]

@ncoghlan I pushed a change addressing the issues above. Do you have any comments on them?

[ncoghlan]

LGTM! @warsaw, over to you for the final review & merge :)

[warsaw]

Looks great! Thanks everyone, for working together on this important update.

[hroncok]

Thank you!