From 68475c04e44f9ff564c480a2be2c3aef476b809b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Fri, 12 Apr 2019 16:46:02 +0200 Subject: [PATCH 01/16] PEP 394: Distributions can choose what does python mean Co-Authored-By: Petr Viktorin --- pep-0394.txt | 222 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 144 insertions(+), 78 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 8da9fb58f53..28b37dd9f51 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -5,12 +5,14 @@ Last-Modified: $Date$ Author: Kerrick Staley , Nick Coghlan , Barry Warsaw , - Petr Viktorin + Petr Viktorin , + Miro Hrončok , Status: Active Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 -Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018 +Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018, + 12-Apr-2019 Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.html @@ -18,91 +20,152 @@ Abstract ======== This PEP provides a convention to ensure that Python scripts can continue to -be portable across ``*nix`` systems, regardless of the default version of the -Python interpreter (i.e. the version invoked by the ``python`` command). +be portable across ``*nix`` systems, regardless of the version of the Python +interpreter invoked by the ``python`` command. * ``python2`` will refer to some version of Python 2.x. * ``python3`` will refer to some version of Python 3.x. -* for the time being, all distributions *should* ensure that ``python``, - if installed, refers to the same target as ``python2``, unless the user - deliberately overrides this or a virtual environment is active. -* however, end users should be aware that ``python`` refers to ``python3`` - on at least Arch Linux (that change is what prompted the creation of this - PEP), so ``python`` should be used in the shebang line only for scripts - that are source compatible with both Python 2 and 3. -* in preparation for an eventual change in the default version of Python, - Python 2 only scripts should either be updated to be source compatible - with Python 3 or else to use ``python2`` in the shebang line. - +* Distributions can choose whether ``python`` is installed and, if it is, + whether it refers to the same target as ``python2`` or ``python3``. +* End users should be aware that the ``python`` command is inconsistent + across Unix-like systems. +* In activated virtual environments, ``python`` always refers to the + environment's Python version. +* Upstream projects should prefer the ``python3`` shebang even if they support + both Python 3 and Python 2. +* Upstream projects only supporting Python 2 should use the ``python2`` + shebang. Recommendation ============== -* Unix-like software distributions (including systems like Mac OS X and +For distributors +---------------- + +* Unix-like software distributions (including systems like macOS and Cygwin) should install the ``python2`` command into the default path whenever a version of the Python 2 interpreter is installed, and the same for ``python3`` and the Python 3 interpreter. * When invoked, ``python2`` should run some version of the Python 2 interpreter, and ``python3`` should run some version of the Python 3 interpreter. -* If the ``python`` command is installed, it should invoke the same version of - Python as the ``python2`` command (however, note that some distributions - have already chosen to have ``python`` implement the ``python3`` - command; see the `Rationale`_ and `Migration Notes`_ below). -* The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should - likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``, - with the original commands invoking these versions by default, but possibly - invoking the Python 3.x versions instead if configured to do so by the +* If the ``python`` command is installed, it should either invoke the same + version of Python as the ``python3`` command or as the ``python2`` + command. +* Distributors may choose to set the behavior of the ``python`` command, + not provide it at all, or make it configurable by the user or system administrator. -* In order to tolerate differences across platforms, all new code that needs +* The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should + likewise be available as ``idle3``, ``pydoc3``, and ``python3-config`` + (and similarly for Python 2.x). The original commands should either invoke + the same version of Python as the ``python`` command, or not be available + at all. +* When packaging software, distributors are encouraged to change less specific + shebangs to more specific ones. This ensures software is used with the latest + version of Python available, and it can remove a dependency on Python 2. + The specifics are left on the distributors. + Examples of this include changing ``python`` shebangs to ``python3`` + when Python 3.x is supported, changing ``python`` shebangs to ``python2`` + when Python 3.x is not yet supported, or changing ``python3`` shebangs to + ``python3.8`` if the software is built with Python 3.8. +* When a virtual environment (created by the PEP 405 ``venv`` package or a + similar tool such as ``virtualenv`` or ``conda``) is active, the ``python`` + command should refer to the virtual environment's interpreter and should + always be available. + The ``python3`` or ``python2`` command (according to the environment's + interpreter) should also be available. + +For developers +-------------- + +* In order to tolerate differences across platforms, new code that needs to invoke the Python interpreter should not specify ``python``, but rather - should specify either ``python2`` or ``python3`` (or the more specific - ``python2.x`` and ``python3.x`` versions; see the `Migration Notes`_). + should specify either ``python3`` or ``python2`` (or the more specific + ``python3.x`` and ``python2.x`` versions; see the `Migration Notes`_). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context. -* One exception to this is scripts that are deliberately written to be source - compatible with both Python 2.x and 3.x. Such scripts may continue to use - ``python`` on their shebang line. -* When packaging software that is source compatible with both versions, - distributions may change such ``python`` shebangs to ``python3``. - This ensures software is used with the latest version of - Python available, and it can remove a dependency on Python 2. * When reinvoking the interpreter from a Python script, querying ``sys.executable`` to avoid hardcoded assumptions regarding the interpreter location remains the preferred approach. -* In controlled environments aimed at expert users, where being explicit - is valued over user experience (for example, in test environments and - package build systems), distributions may choose to not provide the - ``python`` command even if ``python2`` is available. - (All software in such a controlled environment must use ``python3`` or - ``python2`` rather than ``python``, which means scripts that deliberately - use ``python`` need to be modified for such environments.) -* When a virtual environment (created by the PEP 405 ``venv`` package or a - similar tool) is active, the ``python`` command should refer to the - virtual environment's interpreter. In other words, activating a virtual - environment counts as deliberate user action to change the default - ``python`` interpreter. +* Scripts that are deliberately written to be source compatible with both + Python 3.x and 2.x should use ``python3`` on their shebang line + (see `Rationale`_ for details). +* One exception to this is scripts that are deliberately written to be used + on an “*old system*” where ``python3`` (or even ``python2``) is not available + (such as the default Python installed on macOS or RHEL 6). + Such scripts may continue to use ``python`` on their shebang line. +* Scripts targeting both “*old systems*” and systems without the default + ``python`` command need to make a compromise and document this situation. + Avoiding shebangs (via the console_scripts Entry Points ([9]_) or similar + means) is the recommended workaround for this problem. +* Applications designed exclusively for a specific environment (such as + a container or virtual environment) may continue to use the ``python`` + command name. These recommendations are the outcome of the relevant python-dev discussions in March and July 2011 ([1]_, [2]_), February 2012 ([4]_), -September 2014 ([6]_), and discussion on GitHub in April 2018 ([7]_). +September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_) +and on python-dev in February 2019 ([8]_). + -Rationale -========= +History of this PEP +=================== -This recommendation is needed as, even though the majority of distributions -still alias the ``python`` command to Python 2, some now alias it to +In 2011, the majority of distributions +aliased the ``python`` command to Python 2, but some started switching it to Python 3 ([5]_). As some of the former distributions did not provide a ``python2`` command by default, there was previously no way for Python 2 code (or any code that invokes the Python 2 interpreter directly rather than via ``sys.executable``) to reliably run on all Unix-like systems without modification, as the ``python`` command would invoke the wrong interpreter version on some systems, and the ``python2`` command would fail completely -on others. The recommendations in this PEP provide a very simple mechanism +on others. This PEP originally provided a very simple mechanism to restore cross-platform support, with minimal additional work required -on the part of distribution maintainers. +on the part of distribution maintainers. Simplified, the recommendation was: + +1. The ``python`` command was preferred for code compatible with both + Python 2 and 3 (since it was available on all systems, even those that + already aliased it to Python 3). +2. The ``python`` command should always invoke Python 2 (to prevent + hard-to-diagnose errors when Python 2 code is run on Python 3). +3. The ``python2`` and ``python3`` commands should be available to specify + the version explicitly. + +However, these recommendations implicitly assumed that Python 2 would always be +available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404), +distributions are making Python 2 optional or removing entirely. +This means either removing the ``python`` command or switching it to invoke +Python 3, invalidating respectively the first or second recommendation. +Also, some distributors decided that their users are better served by +ignoring the PEP's recommendations, making the PEP's supposedly +cross-platform recommendations on ``python`` and ``python2`` in shebangs +increasingly unreliable. + + +.. _rationale: + +Current Rationale +================= + +As of 2019, nearly all new systems include Python 3 and the ``python3`` +command. This makes the ``python3`` command the best general choice for +code that can run on either Python 3.x or 2.x, even though it is not +available everywhere. + +The recommendation is skewed toward current and future systems, leaving +behind “*old systems*” (like RHEL 6 or default Python installed on macOS). +On these systems, Python software is rarely updated and any recommendations +this PEP makes would likely be ignored. + +Also, since distributors often ignored recommendations the PEP gave +regarding the ``python`` command (for what they saw as legitimate special +needs), this PEP now gives them broad control over the command. +Correspondingly, users are advised to not use the ``python`` command +in cross-platform code. +Instead, this PEP specifies the expected behavior of the ``python3`` and +``python2`` commands, which is not controversial. Future Changes to this Recommendation @@ -111,7 +174,7 @@ Future Changes to this Recommendation This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series -will continue until at least 2020. +will continue until January 2020. Migration Notes @@ -129,27 +192,25 @@ making such a change. and other users. Updating the ``python`` command to invoke ``python3`` by default indicates that a distribution is willing to break such scripts with errors that are potentially quite confusing for users that aren't - yet familiar with the backwards incompatible changes in Python 3. For + familiar with the backwards incompatible changes in Python 3. For example, while the change of ``print`` from a statement to a builtin function is relatively simple for automated converters to handle, the - SyntaxError from attempting to use the Python 2 notation in versions of - Python 3 prior to 3.4.2 is thoroughly confusing if you aren't already - aware of the change:: + SyntaxError from attempting to use the Python 2 notation in Python 3 + may be confusing for users that are not aware of the change:: $ python3 -c 'print "Hello, world!"' File "", line 1 print "Hello, world!" - ^ - SyntaxError: invalid syntax - - (In Python 3.4.2+, that generic error message has been replaced with the - more explicit "SyntaxError: Missing parentheses in call to 'print'") -* Avoiding breakage of such third party scripts is the key reason this - PEP recommends that ``python`` continue to refer to ``python2`` for the - time being. Until the conventions described in this PEP are more widely - adopted, having ``python`` invoke ``python2`` will remain the recommended - option. -* The ``pythonX.X`` (e.g. ``python2.6``) commands exist on some systems, on + ^ + SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello, world!")? + + While this might be obvious for experienced Pythonistas, such scripts + might even be run by people who are not familiar with Python at all. + Avoiding breakage of such third party scripts was the key reason this + PEP used to recommend that ``python`` continue to refer to ``python2``. +* The error message ``python: command not found`` tends to be surprisingly + actionable, even for people unfamiliar with Python. +* The ``pythonX.X`` (e.g. ``python3.6``) commands exist on modern systems, on which they invoke specific minor versions of the Python interpreter. It can be useful for distribution-specific packages to take advantage of these utilities if they exist, since it will prevent code breakage if the default @@ -162,8 +223,8 @@ making such a change. * When the ``pythonX.X`` binaries are provided by a distribution, the ``python2`` and ``python3`` commands should refer to one of those files rather than being provided as a separate binary file. -* It is strongly encouraged that distribution-specific packages use ``python2`` - or ``python3`` rather than ``python``, even in code that is not intended to +* It is strongly encouraged that distribution-specific packages use ``python3`` + (or ``python2``) rather than ``python``, even in code that is not intended to operate on other distributions. This will reduce problems if the distribution later decides to change the version of the Python interpreter that the ``python`` command invokes, or if a sysadmin installs a custom @@ -175,14 +236,13 @@ making such a change. versa. That way, if a sysadmin does decide to replace the installed ``python`` file, they can do so without inadvertently deleting the previously installed binary. -* If the Python 2 interpreter becomes uncommon, scripts should nevertheless +* When the Python 2 interpreter becomes uncommon, scripts should nevertheless continue to use the ``python3`` convention rather that just ``python``. This will ease transition in the event that yet another major version of Python is released. * If these conventions are adhered to, it will become the case that the ``python`` command is only executed in an interactive manner as a user - convenience, or to run scripts that are source compatible with both Python - 2 and Python 3. + convenience. Backwards Compatibility @@ -194,7 +254,7 @@ these commands. This is mostly a non-issue, since the sysadmin can simply create these symbolic links and avoid further problems. It is a significantly more obvious breakage than the sometimes cryptic errors that can arise when attempting to execute a script containing Python 2 specific syntax with a -Python 3 interpreter. +Python 3 interpreter or vice versa. Application to the CPython Reference Interpreter @@ -209,7 +269,7 @@ items are relative symbolic links):: python -> python2 -> python2.7 python-config -> python2-config -> python2.7-config -Similar adjustments were made to the Mac OS X binary installer. +Similar adjustments were made to the macOS binary installer. This feature first appeared in the default installation process in CPython 2.7.3. @@ -252,8 +312,7 @@ Exclusion of MS Windows This PEP deliberately excludes any proposals relating to Microsoft Windows, as devising an equivalent solution for Windows was deemed too complex to handle here. PEP 397 and the related discussion on the python-dev mailing list -address this issue (like this PEP, the PEP 397 launcher invokes Python 2 by -default if versions of both Python 2 and 3 are installed on the system). +address this issue. References @@ -281,6 +340,13 @@ References minor edits (https://github.com/python/peps/pull/630) +.. [8] Another update for PEP 394 -- The "python" Command on Unix-Like Systems + (https://mail.python.org/pipermail/python-dev/2019-February/156272.html) + +.. [9] The console_scripts Entry Point + (https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point) + + Copyright =========== This document has been placed in the public domain. From abf3bf3f85abc7a74b0deb3166481346b6e0daa9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Fri, 12 Apr 2019 22:42:49 +0200 Subject: [PATCH 02/16] fixup! PEP 394: Distributions can choose what does python mean --- pep-0394.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 28b37dd9f51..34d28279055 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -56,10 +56,10 @@ For distributors not provide it at all, or make it configurable by the user or system administrator. * The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should - likewise be available as ``idle3``, ``pydoc3``, and ``python3-config`` - (and similarly for Python 2.x). The original commands should either invoke - the same version of Python as the ``python`` command, or not be available - at all. + likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``; + Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``. + The original commands should either invoke the same version of Python + as the ``python`` command, or not be available at all. * When packaging software, distributors are encouraged to change less specific shebangs to more specific ones. This ensures software is used with the latest version of Python available, and it can remove a dependency on Python 2. From 4a4cdf7c34bb52a14a8faad858751748b341d709 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Mon, 15 Apr 2019 18:13:44 +0200 Subject: [PATCH 03/16] fixup! PEP 394: Distributions can choose what does python mean --- pep-0394.txt | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 34d28279055..22016639309 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -31,9 +31,9 @@ interpreter invoked by the ``python`` command. across Unix-like systems. * In activated virtual environments, ``python`` always refers to the environment's Python version. -* Upstream projects should prefer the ``python3`` shebang even if they support - both Python 3 and Python 2. -* Upstream projects only supporting Python 2 should use the ``python2`` +* Cross-platform scripts should prefer the ``python3`` shebang even if they + support both Python 3 and Python 2. +* Cross-platform scripts only supporting Python 2 should use the ``python2`` shebang. Recommendation @@ -135,11 +135,11 @@ on the part of distribution maintainers. Simplified, the recommendation was: However, these recommendations implicitly assumed that Python 2 would always be available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404), -distributions are making Python 2 optional or removing entirely. +distributions are making Python 2 optional or removing it entirely. This means either removing the ``python`` command or switching it to invoke Python 3, invalidating respectively the first or second recommendation. Also, some distributors decided that their users are better served by -ignoring the PEP's recommendations, making the PEP's supposedly +ignoring the PEP's recommendations, making the PEP's nominally cross-platform recommendations on ``python`` and ``python2`` in shebangs increasingly unreliable. @@ -242,7 +242,7 @@ making such a change. is released. * If these conventions are adhered to, it will become the case that the ``python`` command is only executed in an interactive manner as a user - convenience. + convenience, or else when using a virtual environment or similar mechanism. Backwards Compatibility From 36e61fab741926628987a9a61f0dc6d5cb623b05 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Thu, 18 Apr 2019 13:48:27 +0200 Subject: [PATCH 04/16] fixup! PEP 394: Distributions can choose what does python mean --- pep-0394.txt | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 22016639309..bc4891db14a 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -236,10 +236,14 @@ making such a change. versa. That way, if a sysadmin does decide to replace the installed ``python`` file, they can do so without inadvertently deleting the previously installed binary. -* When the Python 2 interpreter becomes uncommon, scripts should nevertheless - continue to use the ``python3`` convention rather that just ``python``. This - will ease transition in the event that yet another major version of Python - is released. +* Even as the Python 2 interpreter becomes less common, scripts should + nevertheless continue to use the ``python3`` convention rather than just + ``python``. In the event of a future Python 4.0 release, distributors may + then choose to simply alias ``python3`` to the same binary as ``python4`` + (if the degree of change will be the same as in any other Python feature + release), or else they may choose to maintain parallel ``python3`` and + ``python4`` stacks for a time (depending on the needs of their specific + audience). * If these conventions are adhered to, it will become the case that the ``python`` command is only executed in an interactive manner as a user convenience, or else when using a virtual environment or similar mechanism. From 268e96d192ded6ef8431b58d2df69dbbea26cd18 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Wed, 8 May 2019 06:30:22 +1000 Subject: [PATCH 05/16] PEP 394: Keep version agnostic shebang recommendation This update reverts back to the version agnostic "python" invocation as the default recommendation for developers, and rewords the rest of the PEP accordingly. In particular, it makes it clear that publishers are free to adopt the attitude of "we assume you are using a virtual environment", while at the same time granting the distributors the freedom they need to make software with the expectation work correctly when run directly against a system Python installation. --- pep-0394.txt | 91 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 37 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index bc4891db14a..e736e805d71 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -78,23 +78,42 @@ For distributors For developers -------------- -* In order to tolerate differences across platforms, new code that needs - to invoke the Python interpreter should not specify ``python``, but rather - should specify either ``python3`` or ``python2`` (or the more specific - ``python3.x`` and ``python2.x`` versions; see the `Migration Notes`_). - This distinction should be made in shebangs, when invoking from a shell - script, when invoking via the system() call, or when invoking in any other - context. * When reinvoking the interpreter from a Python script, querying ``sys.executable`` to avoid hardcoded assumptions regarding the interpreter location remains the preferred approach. -* Scripts that are deliberately written to be source compatible with both - Python 3.x and 2.x should use ``python3`` on their shebang line - (see `Rationale`_ for details). -* One exception to this is scripts that are deliberately written to be used - on an “*old system*” where ``python3`` (or even ``python2``) is not available - (such as the default Python installed on macOS or RHEL 6). - Such scripts may continue to use ``python`` on their shebang line. +* While far from being universally available, ``python`` remains the + preferred spelling for explicitly invoking Python, as this is the + spelling that virtual environments make consistently available + across different platforms and Python installations. +* In shebang lines, the preferred spelling is ``/usr/bin/env python``, + as this instructs the script to respect the active virtual environment. +* For Python 3 only scripts that do not support being executed on Python + 2 at all, it is recommended to instead use ``python3`` and + ``/usr/bin/env python3``, as these will never invoke Python 2, and are + expected to work for both Python 3 virtual environments and Python 3 + system installations +* For Python 2 only scripts that do not support being executed on Python + 3 at all, it is recommended to instead use ``python2`` and + ``/usr/bin/env python2``, as these will never invoke Python 3, and are + expected to work for Python 2 virtual environments and at least some + Python 2 system installations. +* In cases where the script is expected to be executed outside virtual + environments, developers will need to be aware of the following + discrepancies across platforms and installation methods: + + * Older Linux distributions will provide a ``python`` command that + refers to Python 2. Most of these distros do *not* provide a + ``python2`` command. + * Some newer Linux distributions will provide a ``python`` command that + refers to Python 3 + * Some Linux distributions will not provide a ``python`` command at + all by default, but will provide a ``python3`` command by default + +* When potentially targeting these environments, developers may either + use a Python package installation tool that rewrites shebang lines for + the installed environment, provide instructions on updating shebang lines + interactively, or else use more specific shebang lines that are + tailored to the target environment. * Scripts targeting both “*old systems*” and systems without the default ``python`` command need to make a compromise and document this situation. Avoiding shebangs (via the console_scripts Entry Points ([9]_) or similar @@ -105,9 +124,9 @@ For developers These recommendations are the outcome of the relevant python-dev discussions in March and July 2011 ([1]_, [2]_), February 2012 ([4]_), -September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_) -and on python-dev in February 2019 ([8]_). - +September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_), +on python-dev in February 2019 ([8]_), and during the PEP update review +in May 2019 ([10]_). History of this PEP @@ -137,11 +156,10 @@ However, these recommendations implicitly assumed that Python 2 would always be available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404), distributions are making Python 2 optional or removing it entirely. This means either removing the ``python`` command or switching it to invoke -Python 3, invalidating respectively the first or second recommendation. -Also, some distributors decided that their users are better served by -ignoring the PEP's recommendations, making the PEP's nominally -cross-platform recommendations on ``python`` and ``python2`` in shebangs -increasingly unreliable. +Python 3. Some distributors also decided that their users were better served by +ignoring the PEP's recommendations, and provided system administrators with the +freedom to configure their systems based on the needs of their particular +environment. .. _rationale: @@ -149,23 +167,19 @@ increasingly unreliable. Current Rationale ================= -As of 2019, nearly all new systems include Python 3 and the ``python3`` -command. This makes the ``python3`` command the best general choice for -code that can run on either Python 3.x or 2.x, even though it is not -available everywhere. +As of 2019, activating a Python virtual environment (or its functional +equivalent) is the best way to obtain a consistent cross-platform and +cross-distribution experience. -The recommendation is skewed toward current and future systems, leaving -behind “*old systems*” (like RHEL 6 or default Python installed on macOS). -On these systems, Python software is rarely updated and any recommendations -this PEP makes would likely be ignored. +Accordingly, it is entirely reasonable for publishers to expect the +availability of such an environment to get their software working +correctly, and push responsibility for handling other environments +(such as system Python installations) onto consumers of the software. -Also, since distributors often ignored recommendations the PEP gave -regarding the ``python`` command (for what they saw as legitimate special -needs), this PEP now gives them broad control over the command. -Correspondingly, users are advised to not use the ``python`` command -in cross-platform code. -Instead, this PEP specifies the expected behavior of the ``python3`` and -``python2`` commands, which is not controversial. +As part of that however, it is also appropriate to give Python +distributors the flexibility they need in order to make the +behaviour of their systems as similar as possible to the behaviour +of an activated virtual environment. Future Changes to this Recommendation @@ -350,6 +364,9 @@ References .. [9] The console_scripts Entry Point (https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point) +.. [10] May 2019 PEP update review + (https://github.com/python/peps/pull/989) + Copyright =========== From 5e439bd0362933db522577fb970f48ef6596fede Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Mon, 20 May 2019 17:14:09 +0200 Subject: [PATCH 06/16] Apply suggestions from code review Co-Authored-By: Carol Willing --- pep-0394.txt | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index e736e805d71..c06f4f5a9f8 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -39,31 +39,38 @@ interpreter invoked by the ``python`` command. Recommendation ============== + +Our recommendations are detailed below. We call out any expectations that these recommendations are based upon. + For distributors ---------------- -* Unix-like software distributions (including systems like macOS and +* We expect Unix-like software distributions (including systems like macOS and Cygwin) should install the ``python2`` command into the default path whenever a version of the Python 2 interpreter is installed, and the same for ``python3`` and the Python 3 interpreter. * When invoked, ``python2`` should run some version of the Python 2 interpreter, and ``python3`` should run some version of the Python 3 interpreter. -* If the ``python`` command is installed, it should either invoke the same +* If the ``python`` command is installed, it is expected to invoke either the same version of Python as the ``python3`` command or as the ``python2`` command. -* Distributors may choose to set the behavior of the ``python`` command, +* Distributors may choose to set the behavior of the ``python`` command as follows: + - ``python2``, + - ``python3``, + - not provide ``python`` command + - allow ``python`` to be configurable by an end user or a system administrator. not provide it at all, or make it configurable by the user or system administrator. * The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``; Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``. - The original commands should either invoke the same version of Python + The commands with no version number should either invoke the same version of Python as the ``python`` command, or not be available at all. -* When packaging software, distributors are encouraged to change less specific +* When packaging third party Python scripts, distributors are encouraged to change less specific shebangs to more specific ones. This ensures software is used with the latest version of Python available, and it can remove a dependency on Python 2. - The specifics are left on the distributors. + The details on what specifics to set are left to the distributors; though, example specifics could include: Examples of this include changing ``python`` shebangs to ``python3`` when Python 3.x is supported, changing ``python`` shebangs to ``python2`` when Python 3.x is not yet supported, or changing ``python3`` shebangs to @@ -73,7 +80,7 @@ For distributors command should refer to the virtual environment's interpreter and should always be available. The ``python3`` or ``python2`` command (according to the environment's - interpreter) should also be available. + interpreter version) should also be available. For developers -------------- @@ -168,10 +175,10 @@ Current Rationale ================= As of 2019, activating a Python virtual environment (or its functional -equivalent) is the best way to obtain a consistent cross-platform and +equivalent) prior to script execution is one way to obtain a consistent cross-platform and cross-distribution experience. -Accordingly, it is entirely reasonable for publishers to expect the +Accordingly, publishers can expect users of the software to provide a suitable execution environment. availability of such an environment to get their software working correctly, and push responsibility for handling other environments (such as system Python installations) onto consumers of the software. From 2e763f35601b29381645b2b25c8c2f3c9b06a731 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Mon, 20 May 2019 17:20:58 +0200 Subject: [PATCH 07/16] Moving letters around --- pep-0394.txt | 58 +++++++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index c06f4f5a9f8..604c252cf6a 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -40,7 +40,8 @@ Recommendation ============== -Our recommendations are detailed below. We call out any expectations that these recommendations are based upon. +Our recommendations are detailed below. +We call out any expectations that these recommendations are based upon. For distributors ---------------- @@ -52,29 +53,36 @@ For distributors * When invoked, ``python2`` should run some version of the Python 2 interpreter, and ``python3`` should run some version of the Python 3 interpreter. -* If the ``python`` command is installed, it is expected to invoke either the same - version of Python as the ``python3`` command or as the ``python2`` +* If the ``python`` command is installed, it is expected to invoke either + the same version of Python as the ``python3`` command or as the ``python2`` command. -* Distributors may choose to set the behavior of the ``python`` command as follows: - - ``python2``, - - ``python3``, - - not provide ``python`` command - - allow ``python`` to be configurable by an end user or a system administrator. - not provide it at all, or make it configurable by the user or - system administrator. +* Distributors may choose to set the behavior of the ``python`` command + as follows: + + * ``python2``, + * ``python3``, + * not provide ``python`` command, + * allow ``python`` to be configurable by an end user or + a system administrator. + * The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``; Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``. - The commands with no version number should either invoke the same version of Python - as the ``python`` command, or not be available at all. -* When packaging third party Python scripts, distributors are encouraged to change less specific - shebangs to more specific ones. This ensures software is used with the latest - version of Python available, and it can remove a dependency on Python 2. - The details on what specifics to set are left to the distributors; though, example specifics could include: - Examples of this include changing ``python`` shebangs to ``python3`` - when Python 3.x is supported, changing ``python`` shebangs to ``python2`` - when Python 3.x is not yet supported, or changing ``python3`` shebangs to - ``python3.8`` if the software is built with Python 3.8. + The commands with no version number should either invoke the same version + of Python as the ``python`` command, or not be available at all. +* When packaging third party Python scripts, distributors are encouraged to + change less specific shebangs to more specific ones. + This ensures software is used with the latest version of Python available, + and it can remove a dependency on Python 2. + The details on what specifics to set are left to the distributors; + though, example specifics could include: + + * Changing ``python`` shebangs to ``python3`` when Python 3.x is supported. + * Changing ``python`` shebangs to ``python2`` when Python 3.x is not yet + supported. + * Changing ``python3`` shebangs to ``python3.8`` if the software is built + with Python 3.8. + * When a virtual environment (created by the PEP 405 ``venv`` package or a similar tool such as ``virtualenv`` or ``conda``) is active, the ``python`` command should refer to the virtual environment's interpreter and should @@ -175,13 +183,11 @@ Current Rationale ================= As of 2019, activating a Python virtual environment (or its functional -equivalent) prior to script execution is one way to obtain a consistent cross-platform and -cross-distribution experience. +equivalent) prior to script execution is one way to obtain a consistent +cross-platform and cross-distribution experience. -Accordingly, publishers can expect users of the software to provide a suitable execution environment. -availability of such an environment to get their software working -correctly, and push responsibility for handling other environments -(such as system Python installations) onto consumers of the software. +Accordingly, publishers can expect users of the software to provide a suitable +execution environment. As part of that however, it is also appropriate to give Python distributors the flexibility they need in order to make the From 627ae44259b937bc938a24305ae3b0a7697ff7e1 Mon Sep 17 00:00:00 2001 From: Carol Willing Date: Mon, 20 May 2019 17:31:27 +0200 Subject: [PATCH 08/16] Change the abstract --- pep-0394.txt | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 604c252cf6a..a9b25f0a4c6 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -7,6 +7,7 @@ Author: Kerrick Staley , Barry Warsaw , Petr Viktorin , Miro Hrončok , + Carol Willing , Status: Active Type: Informational Content-Type: text/x-rst @@ -19,22 +20,25 @@ Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.ht Abstract ======== -This PEP provides a convention to ensure that Python scripts can continue to -be portable across ``*nix`` systems, regardless of the version of the Python -interpreter invoked by the ``python`` command. - -* ``python2`` will refer to some version of Python 2.x. -* ``python3`` will refer to some version of Python 3.x. -* Distributions can choose whether ``python`` is installed and, if it is, - whether it refers to the same target as ``python2`` or ``python3``. -* End users should be aware that the ``python`` command is inconsistent - across Unix-like systems. -* In activated virtual environments, ``python`` always refers to the - environment's Python version. -* Cross-platform scripts should prefer the ``python3`` shebang even if they - support both Python 3 and Python 2. -* Cross-platform scripts only supporting Python 2 should use the ``python2`` - shebang. +This PEP outlines the behavior of Python scripts when the ``python`` command +is invoked. +Depending on a distribution or system configuration, +``python`` may or may not be installed. +If ``python`` is installed its target interpreter may refer to ``python2`` +or ``python3``. +End users may be unaware of this inconsistency across Unix-like systems. +This PEP's goal is to reduce user confusion about what ``python`` references +and what will be the script's behavior. + +The recommendations in the next section of this PEP will outline the behavior +when: + +* using virtual environments +* cross-platform scripts with shebangs for either ``python2`` or ``python3`` + +The PEP's goal is to clarify the behavior for script end users, distribution +providers, and script maintainers / authors. + Recommendation ============== From 4af8f147d56c819d665cfbfde5cf01adb19b8b4b Mon Sep 17 00:00:00 2001 From: Carol Willing Date: Mon, 20 May 2019 17:35:09 +0200 Subject: [PATCH 09/16] Remove a paragraph --- pep-0394.txt | 5 ----- 1 file changed, 5 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index a9b25f0a4c6..56463dfaaa0 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -193,11 +193,6 @@ cross-platform and cross-distribution experience. Accordingly, publishers can expect users of the software to provide a suitable execution environment. -As part of that however, it is also appropriate to give Python -distributors the flexibility they need in order to make the -behaviour of their systems as similar as possible to the behaviour -of an activated virtual environment. - Future Changes to this Recommendation ===================================== From 8d88ea04d3d7fbf04df0c1c14194ee06fd3df34b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Mon, 20 May 2019 17:38:31 +0200 Subject: [PATCH 10/16] ValueError: some authors have more than one email address listed --- pep-0394.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0394.txt b/pep-0394.txt index 56463dfaaa0..1d5fab14ab9 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -7,7 +7,7 @@ Author: Kerrick Staley , Barry Warsaw , Petr Viktorin , Miro Hrončok , - Carol Willing , + Carol Willing , Status: Active Type: Informational Content-Type: text/x-rst From 9546b158da2567d54f2bc7c4be0df8168f576686 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Wed, 22 May 2019 16:23:28 +0200 Subject: [PATCH 11/16] Apply suggestions from code review Co-Authored-By: Petr Viktorin --- pep-0394.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 1d5fab14ab9..e7eb2092135 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -34,7 +34,7 @@ The recommendations in the next section of this PEP will outline the behavior when: * using virtual environments -* cross-platform scripts with shebangs for either ``python2`` or ``python3`` +* writing cross-platform scripts with shebangs for either ``python2`` or ``python3`` The PEP's goal is to clarify the behavior for script end users, distribution providers, and script maintainers / authors. @@ -79,7 +79,7 @@ For distributors This ensures software is used with the latest version of Python available, and it can remove a dependency on Python 2. The details on what specifics to set are left to the distributors; - though, example specifics could include: + though. Example specifics could include: * Changing ``python`` shebangs to ``python3`` when Python 3.x is supported. * Changing ``python`` shebangs to ``python2`` when Python 3.x is not yet From fee50a41a70cdac33fb64fbcfc4a6f24c11a82ef Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Fri, 24 May 2019 14:58:27 +0200 Subject: [PATCH 12/16] Address review comments * Rename section headers * Push script publishers and users to virtual environments * Rewording --- pep-0394.txt | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index e7eb2092135..cf9ce994494 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -47,8 +47,8 @@ Recommendation Our recommendations are detailed below. We call out any expectations that these recommendations are based upon. -For distributors ----------------- +For Python runtime distributors +------------------------------- * We expect Unix-like software distributions (including systems like macOS and Cygwin) should install the ``python2`` command into the default path @@ -94,16 +94,15 @@ For distributors The ``python3`` or ``python2`` command (according to the environment's interpreter version) should also be available. -For developers --------------- +For Python script publishers +---------------------------- * When reinvoking the interpreter from a Python script, querying ``sys.executable`` to avoid hardcoded assumptions regarding the interpreter location remains the preferred approach. -* While far from being universally available, ``python`` remains the - preferred spelling for explicitly invoking Python, as this is the - spelling that virtual environments make consistently available - across different platforms and Python installations. +* Encourage your end users to use a virtual environment. + This makes the user's environment more predictable (possibly resulting + in fewer issues), and helps avoid disrupting their system. * In shebang lines, the preferred spelling is ``/usr/bin/env python``, as this instructs the script to respect the active virtual environment. * For Python 3 only scripts that do not support being executed on Python @@ -141,6 +140,19 @@ For developers a container or virtual environment) may continue to use the ``python`` command name. +For end users of Python +----------------------- + +* While far from being universally available, ``python`` remains the + preferred spelling for explicitly invoking Python, as this is the + spelling that virtual environments make consistently available + across different platforms and Python installations. +* For software that is not distributed with (or developed for) your system, + we recommend using a virtual environment, possibly with an environment + manager like ``conda`` or ``pipenv``, to help avoid disrupting your system + Python installation. + + These recommendations are the outcome of the relevant python-dev discussions in March and July 2011 ([1]_, [2]_), February 2012 ([4]_), September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_), @@ -176,9 +188,9 @@ available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404), distributions are making Python 2 optional or removing it entirely. This means either removing the ``python`` command or switching it to invoke Python 3. Some distributors also decided that their users were better served by -ignoring the PEP's recommendations, and provided system administrators with the -freedom to configure their systems based on the needs of their particular -environment. +ignoring the PEP's original recommendations, and provided system +administrators with the freedom to configure their systems based on +the needs of their particular environment. .. _rationale: @@ -262,8 +274,8 @@ making such a change. versa. That way, if a sysadmin does decide to replace the installed ``python`` file, they can do so without inadvertently deleting the previously installed binary. -* Even as the Python 2 interpreter becomes less common, scripts should - nevertheless continue to use the ``python3`` convention rather than just +* Even as the Python 2 interpreter becomes less common, it remains reasonable + for scripts to continue to use the ``python3`` convention, rather than just ``python``. In the event of a future Python 4.0 release, distributors may then choose to simply alias ``python3`` to the same binary as ``python4`` (if the degree of change will be the same as in any other Python feature From 4b20f8c130df2c6b9b1a9900ea41eca51999ee05 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Wed, 26 Jun 2019 08:27:47 +1000 Subject: [PATCH 13/16] Remove "preferred" shebang line wording --- pep-0394.txt | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index cf9ce994494..6171666f2b0 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -103,18 +103,9 @@ For Python script publishers * Encourage your end users to use a virtual environment. This makes the user's environment more predictable (possibly resulting in fewer issues), and helps avoid disrupting their system. -* In shebang lines, the preferred spelling is ``/usr/bin/env python``, +* For scripts that are only expected to be run in an activated virtual + environment, shebang lines can be written as ``/usr/bin/env python``, as this instructs the script to respect the active virtual environment. -* For Python 3 only scripts that do not support being executed on Python - 2 at all, it is recommended to instead use ``python3`` and - ``/usr/bin/env python3``, as these will never invoke Python 2, and are - expected to work for both Python 3 virtual environments and Python 3 - system installations -* For Python 2 only scripts that do not support being executed on Python - 3 at all, it is recommended to instead use ``python2`` and - ``/usr/bin/env python2``, as these will never invoke Python 3, and are - expected to work for Python 2 virtual environments and at least some - Python 2 system installations. * In cases where the script is expected to be executed outside virtual environments, developers will need to be aware of the following discrepancies across platforms and installation methods: From cca28c03fbc1c08ddfaf20e43c8ad3ad8ae7155b Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Wed, 26 Jun 2019 08:33:17 +1000 Subject: [PATCH 14/16] Change post date --- pep-0394.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0394.txt b/pep-0394.txt index 6171666f2b0..62cdceef801 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -13,7 +13,7 @@ Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018, - 12-Apr-2019 + 26-Jun-2019 Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.html From 5d3e9d8f9ee7a23f25ef62b8ab5ee54ae1c603e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= Date: Fri, 28 Jun 2019 16:07:21 +0200 Subject: [PATCH 15/16] Little tweaks --- pep-0394.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 62cdceef801..8dd83ee2de9 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -51,7 +51,7 @@ For Python runtime distributors ------------------------------- * We expect Unix-like software distributions (including systems like macOS and - Cygwin) should install the ``python2`` command into the default path + Cygwin) to install the ``python2`` command into the default path whenever a version of the Python 2 interpreter is installed, and the same for ``python3`` and the Python 3 interpreter. * When invoked, ``python2`` should run some version of the Python 2 @@ -104,7 +104,7 @@ For Python script publishers This makes the user's environment more predictable (possibly resulting in fewer issues), and helps avoid disrupting their system. * For scripts that are only expected to be run in an activated virtual - environment, shebang lines can be written as ``/usr/bin/env python``, + environment, shebang lines can be written as ``#!/usr/bin/env python``, as this instructs the script to respect the active virtual environment. * In cases where the script is expected to be executed outside virtual environments, developers will need to be aware of the following @@ -114,9 +114,9 @@ For Python script publishers refers to Python 2. Most of these distros do *not* provide a ``python2`` command. * Some newer Linux distributions will provide a ``python`` command that - refers to Python 3 + refers to Python 3. * Some Linux distributions will not provide a ``python`` command at - all by default, but will provide a ``python3`` command by default + all by default, but will provide a ``python3`` command by default. * When potentially targeting these environments, developers may either use a Python package installation tool that rewrites shebang lines for @@ -148,7 +148,7 @@ These recommendations are the outcome of the relevant python-dev discussions in March and July 2011 ([1]_, [2]_), February 2012 ([4]_), September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_), on python-dev in February 2019 ([8]_), and during the PEP update review -in May 2019 ([10]_). +in May/June 2019 ([10]_). History of this PEP From 6dbfc2f6e64adb4a970526d9f6ba46656f7a1c23 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 3 Jul 2019 17:14:48 +0200 Subject: [PATCH 16/16] Address review comments - Clarify wording on "older" Linux distros - Remove discussion aroung Python 4.0 --- pep-0394.txt | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/pep-0394.txt b/pep-0394.txt index 8dd83ee2de9..c2c83d57e87 100644 --- a/pep-0394.txt +++ b/pep-0394.txt @@ -111,8 +111,7 @@ For Python script publishers discrepancies across platforms and installation methods: * Older Linux distributions will provide a ``python`` command that - refers to Python 2. Most of these distros do *not* provide a - ``python2`` command. + refers to Python 2, and will likely not provide a ``python2`` command. * Some newer Linux distributions will provide a ``python`` command that refers to Python 3. * Some Linux distributions will not provide a ``python`` command at @@ -267,12 +266,7 @@ making such a change. previously installed binary. * Even as the Python 2 interpreter becomes less common, it remains reasonable for scripts to continue to use the ``python3`` convention, rather than just - ``python``. In the event of a future Python 4.0 release, distributors may - then choose to simply alias ``python3`` to the same binary as ``python4`` - (if the degree of change will be the same as in any other Python feature - release), or else they may choose to maintain parallel ``python3`` and - ``python4`` stacks for a time (depending on the needs of their specific - audience). + ``python``. * If these conventions are adhered to, it will become the case that the ``python`` command is only executed in an interactive manner as a user convenience, or else when using a virtual environment or similar mechanism.