PEP 394: Allow for more flexibility in handling /usr/bin/python

pep-0394.txt

@@ -5,104 +5,205 @@ Last-Modified: $Date$
 Author: Kerrick Staley <mail@kerrickstaley.com>,
         Nick Coghlan <ncoghlan@gmail.com>,
         Barry Warsaw <barry@python.org>,
-        Petr Viktorin <encukou@gmail.com>
+        Petr Viktorin <encukou@gmail.com>,
+        Miro Hrončok <miro@hroncok.cz>,
+        Carol Willing <willingc@gmail.com>,
 Status: Active
 Type: Informational
 Content-Type: text/x-rst
 Created: 02-Mar-2011
-Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018
+Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018,
+              12-Apr-2019
 Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.html
 
 
 Abstract
 ========
 
-This PEP provides a convention to ensure that Python scripts can continue to
-be portable across ``*nix`` systems, regardless of the default version of the
-Python interpreter (i.e. the version invoked by the ``python`` command).
+This PEP outlines the behavior of Python scripts when the ``python`` command
+is invoked.
+Depending on a distribution or system configuration,
+``python`` may or may not be installed.
+If ``python`` is installed its target interpreter may refer to ``python2``
+or ``python3``.
+End users may be unaware of this inconsistency across Unix-like systems.
+This PEP's goal is to reduce user confusion about what ``python`` references
+and what will be the script's behavior.
 
-* ``python2`` will refer to some version of Python 2.x.
-* ``python3`` will refer to some version of Python 3.x.
-* for the time being, all distributions *should* ensure that ``python``,
-  if installed, refers to the same target as ``python2``, unless the user
-  deliberately overrides this or a virtual environment is active.
-* however, end users should be aware that ``python`` refers to ``python3``
-  on at least Arch Linux (that change is what prompted the creation of this
-  PEP), so ``python`` should be used in the shebang line only for scripts
-  that are source compatible with both Python 2 and 3.
-* in preparation for an eventual change in the default version of Python,
-  Python 2 only scripts should either be updated to be source compatible
-  with Python 3 or else to use ``python2`` in the shebang line.
+The recommendations in the next section of this PEP will outline the behavior
+when:
+
+* using virtual environments
+* writing cross-platform scripts with shebangs for either ``python2`` or ``python3``
+
+The PEP's goal is to clarify the behavior for script end users, distribution
+providers, and script maintainers / authors.
 
 
 Recommendation
 ==============
 
-* Unix-like software distributions (including systems like Mac OS X and
+
+Our recommendations are detailed below.
+We call out any expectations that these recommendations are based upon.
+
+For Python runtime distributors
+-------------------------------
+
+* We expect Unix-like software distributions (including systems like macOS and
   Cygwin) should install the ``python2`` command into the default path
   whenever a version of the Python 2 interpreter is installed, and the same
   for ``python3`` and the Python 3 interpreter.
 * When  invoked, ``python2`` should run some version of the Python 2
   interpreter, and ``python3`` should run some version of the Python 3
   interpreter.
-* If the ``python`` command is installed, it should invoke the same version of
-  Python as the ``python2`` command (however, note that some distributions
-  have already chosen to have ``python`` implement the ``python3``
-  command; see the `Rationale`_ and `Migration Notes`_ below).
-* The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should
-  likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``,
-  with the original commands invoking these versions by default, but possibly
-  invoking the Python 3.x versions instead if configured to do so by the
-  system administrator.
-* In order to tolerate differences across platforms, all new code that needs
-  to invoke the Python interpreter should not specify ``python``, but rather
-  should specify either ``python2`` or ``python3`` (or the more specific
-  ``python2.x`` and ``python3.x`` versions; see the `Migration Notes`_).
-  This distinction should be made in shebangs, when invoking from a shell
-  script, when invoking via the system() call, or when invoking in any other
-  context.
-* One exception to this is scripts that are deliberately written to be source
-  compatible with both Python 2.x and 3.x. Such scripts may continue to use
-  ``python`` on their shebang line.
-* When packaging software that is source compatible with both versions,
-  distributions may change such ``python`` shebangs to ``python3``.
-  This ensures software is used with the latest version of
-  Python available, and it can remove a dependency on Python 2.
+* If the ``python`` command is installed, it is expected to invoke either
+  the same version of Python as the ``python3`` command or as the ``python2``
+  command.
+* Distributors may choose to set the behavior of the ``python`` command
+  as follows:
+
+   * ``python2``,
+   * ``python3``,
+   * not provide ``python`` command,
+   * allow ``python`` to be configurable by an end user or
+     a system administrator.
+
+* The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should
+  likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``;
+  Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``.
+  The commands with no version number should either invoke the same version
+  of Python as the ``python`` command, or not be available at all.
+* When packaging third party Python scripts, distributors are encouraged to
+  change less specific shebangs to more specific ones.
+  This ensures software is used with the latest version of Python available,
+  and it can remove a dependency on Python 2.
+  The details on what specifics to set are left to the distributors;
+  though. Example specifics could include:
+
+  * Changing ``python`` shebangs to ``python3`` when Python 3.x is supported.
+  * Changing ``python`` shebangs to ``python2`` when Python 3.x is not yet
+    supported.
+  * Changing ``python3`` shebangs to ``python3.8`` if the software is built
+    with Python 3.8.
+
+* When a virtual environment (created by the PEP 405 ``venv`` package or a
+  similar tool such as ``virtualenv`` or ``conda``) is active, the ``python``
+  command should refer to the virtual environment's interpreter and should
+  always be available.
+  The ``python3`` or ``python2`` command (according to the environment's
+  interpreter version) should also be available.
+
+For Python script publishers
+----------------------------
+
 * When reinvoking the interpreter from a Python script, querying
   ``sys.executable`` to avoid hardcoded assumptions regarding the
   interpreter location remains the preferred approach.
-* In controlled environments aimed at expert users, where being explicit
-  is valued over user experience (for example, in test environments and
-  package build systems), distributions may choose to not provide the
-  ``python`` command even if ``python2`` is available.
-  (All software in such a controlled environment must use ``python3`` or
-  ``python2`` rather than ``python``, which means scripts that deliberately
-  use ``python`` need to be modified for such environments.)
-* When a virtual environment (created by the PEP 405 ``venv`` package or a
-  similar tool) is active, the ``python`` command should refer to the
-  virtual environment's interpreter. In other words, activating a virtual
-  environment counts as deliberate user action to change the default
-  ``python`` interpreter.
+* Encourage your end users to use a virtual environment.
+  This makes the user's environment more predictable (possibly resulting
+  in fewer issues), and helps avoid disrupting their system.
+* In shebang lines, the preferred spelling is ``/usr/bin/env python``,
+  as this instructs the script to respect the active virtual environment.
+* For Python 3 only scripts that do not support being executed on Python
+  2 at all, it is recommended to instead use ``python3`` and
+  ``/usr/bin/env python3``, as these will never invoke Python 2, and are
+  expected to work for both Python 3 virtual environments and Python 3
+  system installations
+* For Python 2 only scripts that do not support being executed on Python
+  3 at all, it is recommended to instead use ``python2`` and
+  ``/usr/bin/env python2``, as these will never invoke Python 3, and are
+  expected to work for Python 2 virtual environments and at least some
+  Python 2 system installations.
+* In cases where the script is expected to be executed outside virtual
+  environments, developers will need to be aware of the following
+  discrepancies across platforms and installation methods:
+
+  * Older Linux distributions will provide a ``python`` command that
+    refers to Python 2. Most of these distros do *not* provide a
+    ``python2`` command.
+  * Some newer Linux distributions will provide a ``python`` command that
+    refers to Python 3
+  * Some Linux distributions will not provide a ``python`` command at
+    all by default, but will provide a ``python3`` command by default
+
+* When potentially targeting these environments, developers may either
+  use a Python package installation tool that rewrites shebang lines for
+  the installed environment, provide instructions on updating shebang lines
+  interactively, or else use more specific shebang lines that are
+  tailored to the target environment.
+* Scripts targeting both “*old systems*” and systems without the default
+  ``python`` command need to make a compromise and document this situation.
+  Avoiding shebangs (via the console_scripts Entry Points ([9]_) or similar
+  means) is the recommended workaround for this problem.
+* Applications designed exclusively for a specific environment (such as
+  a container or virtual environment) may continue to use the ``python``
+  command name.
+
+For end users of Python
+-----------------------
+
+* While far from being universally available, ``python`` remains the
+  preferred spelling for explicitly invoking Python, as this is the
+  spelling that virtual environments make consistently available
+  across different platforms and Python installations.
+* For software that is not distributed with (or developed for) your system,
+  we recommend using a virtual environment, possibly with an environment
+  manager like ``conda`` or ``pipenv``, to help avoid disrupting your system
+  Python installation.
+
 
 These recommendations are the outcome of the relevant python-dev discussions
 in March and July 2011 ([1]_, [2]_), February 2012 ([4]_),
-September 2014 ([6]_), and discussion on GitHub in April 2018 ([7]_).
+September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_),
+on python-dev in February 2019 ([8]_), and during the PEP update review
+in May 2019 ([10]_).
 
 
-Rationale
-=========
+History of this PEP
+===================
 
-This recommendation is needed as, even though the majority of distributions
-still alias the ``python`` command to Python 2, some now alias it to
+In 2011, the majority of distributions
+aliased the ``python`` command to Python 2, but some started switching it to
 Python 3 ([5]_). As some of the former distributions did not provide a
 ``python2`` command by default, there was previously no way for Python 2 code
 (or any code that invokes the Python 2 interpreter directly rather than via
 ``sys.executable``) to reliably run on all Unix-like systems without
 modification, as the ``python`` command would invoke the wrong interpreter
 version on some systems, and the ``python2`` command would fail completely
-on others. The recommendations in this PEP provide a very simple mechanism
+on others. This PEP originally provided a very simple mechanism
 to restore cross-platform support, with minimal additional work required
-on the part of distribution maintainers.
+on the part of distribution maintainers. Simplified, the recommendation was:
+
+1. The ``python`` command was preferred for code compatible with both
+   Python 2 and 3 (since it was available on all systems, even those that
+   already aliased it to Python 3).
+2. The ``python`` command should always invoke Python 2 (to prevent
+   hard-to-diagnose errors when Python 2 code is run on Python 3).
+3. The ``python2`` and ``python3`` commands should be available to specify
+   the version explicitly.
+
+However, these recommendations implicitly assumed that Python 2 would always be
+available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404),
+distributions are making Python 2 optional or removing it entirely.
+This means either removing the ``python`` command or switching it to invoke
+Python 3. Some distributors also decided that their users were better served by
+ignoring the PEP's original recommendations, and provided system
+administrators with the freedom to configure their systems based on
+the needs of their particular environment.
+
+
+.. _rationale:
+
+Current Rationale
+=================
+
+As of 2019, activating a Python virtual environment (or its functional
+equivalent) prior to script execution is one way to obtain a consistent
+cross-platform and cross-distribution experience.
+
+Accordingly, publishers can expect users of the software to provide a suitable
+execution environment.
 
 
 Future Changes to this Recommendation
@@ -111,7 +212,7 @@ Future Changes to this Recommendation
 This recommendation will be periodically reviewed over the next few years,
 and updated when the core development team judges it appropriate. As a
 point of reference, regular maintenance releases for the Python 2.7 series
-will continue until at least 2020.
+will continue until January 2020.
 
 
 Migration Notes
@@ -129,27 +230,25 @@ making such a change.
   and other users. Updating the ``python`` command to invoke ``python3``
   by default indicates that a distribution is willing to break such scripts
   with errors that are potentially quite confusing for users that aren't
-  yet familiar with the backwards incompatible changes in Python 3. For
+  familiar with the backwards incompatible changes in Python 3. For
   example, while the change of ``print`` from a statement to a builtin
   function is relatively simple for automated converters to handle, the
-  SyntaxError from attempting to use the Python 2 notation in versions of
-  Python 3 prior to 3.4.2 is thoroughly confusing if you aren't already
-  aware of the change::
+  SyntaxError from attempting to use the Python 2 notation in Python 3
+  may be confusing for users that are not aware of the change::
 
       $ python3 -c 'print "Hello, world!"'
         File "<string>", line 1
           print "Hello, world!"
-                              ^
-      SyntaxError: invalid syntax
-
-  (In Python 3.4.2+, that generic error message has been replaced with the
-  more explicit "SyntaxError: Missing parentheses in call to 'print'")
-* Avoiding breakage of such third party scripts is the key reason this
-  PEP recommends that ``python`` continue to refer to ``python2`` for the
-  time being. Until the conventions described in this PEP are more widely
-  adopted, having ``python`` invoke ``python2`` will remain the recommended
-  option.
-* The ``pythonX.X`` (e.g. ``python2.6``) commands exist on some systems, on
+                ^
+      SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello, world!")?
+
+  While this might be obvious for experienced Pythonistas, such scripts
+  might even be run by people who are not familiar with Python at all.
+  Avoiding breakage of such third party scripts was the key reason this
+  PEP used to recommend that ``python`` continue to refer to ``python2``.
+* The error message ``python: command not found`` tends to be surprisingly
+  actionable, even for people unfamiliar with Python.
+* The ``pythonX.X`` (e.g. ``python3.6``) commands exist on modern systems, on
   which they invoke specific minor versions of the Python interpreter. It
   can be useful for distribution-specific packages to take advantage of these
   utilities if they exist, since it will prevent code breakage if the default
@@ -162,8 +261,8 @@ making such a change.
 * When the ``pythonX.X`` binaries are provided by a distribution, the
   ``python2`` and ``python3`` commands should refer to one of those files
   rather than being provided as a separate binary file.
-* It is strongly encouraged that distribution-specific packages use ``python2``
-  or ``python3`` rather than ``python``, even in code that is not intended to
+* It is strongly encouraged that distribution-specific packages use ``python3``
+  (or ``python2``) rather than ``python``, even in code that is not intended to
   operate on other distributions. This will reduce problems if the
   distribution later decides to change the version of the Python interpreter
   that the ``python`` command invokes, or if a sysadmin installs a custom
@@ -175,14 +274,17 @@ making such a change.
   versa. That way, if a sysadmin does decide to replace the installed
   ``python`` file, they can do so without inadvertently deleting the
   previously installed binary.
-* If the Python 2 interpreter becomes uncommon, scripts should nevertheless
-  continue to use the ``python3`` convention rather that just ``python``. This
-  will ease transition in the event that yet another major version of Python
-  is released.
+* Even as the Python 2 interpreter becomes less common, it remains reasonable
+  for scripts to continue to use the ``python3`` convention, rather than just
+  ``python``. In the event of a future Python 4.0 release, distributors may
+  then choose to simply alias ``python3`` to the same binary as ``python4``
+  (if the degree of change will be the same as in any other Python feature
+  release), or else they may choose to maintain parallel ``python3`` and
+  ``python4`` stacks for a time (depending on the needs of their specific
+  audience).
 * If these conventions are adhered to, it will become the case that the
   ``python`` command is only executed in an interactive manner as a user
-  convenience, or to run scripts that are source compatible with both Python
-  2 and Python 3.
+  convenience, or else when using a virtual environment or similar mechanism.
 
 
 Backwards Compatibility
@@ -194,7 +296,7 @@ these commands. This is mostly a non-issue, since the sysadmin can simply
 create these symbolic links and avoid further problems. It is a significantly
 more obvious breakage than the sometimes cryptic errors that can arise when
 attempting to execute a script containing Python 2 specific syntax with a
-Python 3 interpreter.
+Python 3 interpreter or vice versa.
 
 
 Application to the CPython Reference Interpreter
@@ -209,7 +311,7 @@ items are relative symbolic links)::
     python -> python2 -> python2.7
     python-config -> python2-config -> python2.7-config
 
-Similar adjustments were made to the Mac OS X binary installer.
+Similar adjustments were made to the macOS binary installer.
 
 This feature first appeared in the default installation process in
 CPython 2.7.3.
@@ -252,8 +354,7 @@ Exclusion of MS Windows
 This PEP deliberately excludes any proposals relating to Microsoft Windows, as
 devising an equivalent solution for Windows was deemed too complex to handle
 here. PEP 397 and the related discussion on the python-dev mailing list
-address this issue (like this PEP, the PEP 397 launcher invokes Python 2 by
-default if versions of both Python 2 and 3 are installed on the system).
+address this issue.
 
 
 References
@@ -281,6 +382,16 @@ References
    minor edits
    (https://github.com/python/peps/pull/630)
 
+.. [8] Another update for PEP 394 -- The "python" Command on Unix-Like Systems
+   (https://mail.python.org/pipermail/python-dev/2019-February/156272.html)
+
+.. [9] The console_scripts Entry Point
+   (https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point)
+
+.. [10] May 2019 PEP update review
+   (https://github.com/python/peps/pull/989)
+
+
 Copyright
 ===========
 This document has been placed in the public domain.