Doc improvements (#3363, #3364, #3367)

- Improve docs by avoiding confusion with distutils
- Update dependency management docs
- Update userguide on miscellaneous and extension

changelog.d/3363.doc.rst

@@ -0,0 +1,3 @@
+Rework some documentation pages to de-emphasize ``distutils`` and the history
+of packaging in the Python ecosystem. The focus of these changes is to make the
+documentation easier to read for new users.

changelog.d/3364.doc.rst

@@ -0,0 +1,2 @@
+Update documentation about dependency management, removing mention to
+the deprecated ``dependency_links`` and adding some small improvements.

changelog.d/3367.doc.rst

@@ -0,0 +1,13 @@
+Extracted text about automatic resource extraction and the zip-safe flag
+from ``userguide/miscellaneous`` to ``deprecated/resource_extraction`` and
+``deprecated/zip_safe``.
+
+Extracted text about additional metadata files from
+``userguide/miscellaneous`` into the existing ``userguide/extension``
+document.
+
+Updated ``userguide/extension`` to better reflect the status of the
+setuptools project.
+
+Removed ``userguide/functionalities_rewrite`` (a virtually empty part of the
+docs).

docs/build_meta.rst

@@ -95,6 +95,9 @@ or::
 
     $ pip install dist/meowpkg-0.0.1.tar.gz
 
+
+.. _backend-wrapper:
+
 Dynamic build dependencies and other ``build_meta`` tweaks
 ----------------------------------------------------------
 

docs/conf.py

@@ -178,7 +178,6 @@
 
 # Allow linking objects on other Sphinx sites seamlessly:
 intersphinx_mapping.update(
-    python2=('https://docs.python.org/2', None),
     python=('https://docs.python.org/3', None),
 )
 

docs/deprecated/dependency_links.rst

@@ -0,0 +1,77 @@
+Specifying dependencies that aren't in PyPI via ``dependency_links``
+====================================================================
+
+.. warning::
+    Dependency links support has been dropped by pip starting with version
+    19.0 (released 2019-01-22).
+
+If your project depends on packages that don't exist on PyPI, you *may* still be
+able to depend on them if they are available for download as:
+
+- an egg, in the standard distutils ``sdist`` format,
+- a single ``.py`` file, or
+- a VCS repository (Subversion, Mercurial, or Git).
+
+You need to add some URLs to the ``dependency_links`` argument to ``setup()``.
+
+The URLs must be either:
+
+1. direct download URLs,
+2. the URLs of web pages that contain direct download links, or
+3. the repository's URL
+
+In general, it's better to link to web pages, because it is usually less
+complex to update a web page than to release a new version of your project.
+You can also use a SourceForge ``showfiles.php`` link in the case where a
+package you depend on is distributed via SourceForge.
+
+If you depend on a package that's distributed as a single ``.py`` file, you
+must include an ``"#egg=project-version"`` suffix to the URL, to give a project
+name and version number.  (Be sure to escape any dashes in the name or version
+by replacing them with underscores.)  EasyInstall will recognize this suffix
+and automatically create a trivial ``setup.py`` to wrap the single ``.py`` file
+as an egg.
+
+In the case of a VCS checkout, you should also append ``#egg=project-version``
+in order to identify for what package that checkout should be used. You can
+append ``@REV`` to the URL's path (before the fragment) to specify a revision.
+Additionally, you can also force the VCS being used by prepending the URL with
+a certain prefix. Currently available are:
+
+-  ``svn+URL`` for Subversion,
+-  ``git+URL`` for Git, and
+-  ``hg+URL`` for Mercurial
+
+A more complete example would be:
+
+    ``vcs+proto://host/path@revision#egg=project-version``
+
+Be careful with the version. It should match the one inside the project files.
+If you want to disregard the version, you have to omit it both in the
+``requires`` and in the URL's fragment.
+
+This will do a checkout (or a clone, in Git and Mercurial parlance) to a
+temporary folder and run ``setup.py bdist_egg``.
+
+The ``dependency_links`` option takes the form of a list of URL strings.  For
+example, this will cause a search of the specified page for eggs or source
+distributions, if the package's dependencies aren't already installed:
+
+.. tab:: setup.cfg
+
+    .. code-block:: ini
+
+        [options]
+        #...
+        dependency_links = http://peak.telecommunity.com/snapshots/
+
+.. tab:: setup.py
+
+    .. code-block:: python
+
+        setup(
+            ...,
+            dependency_links=[
+                "http://peak.telecommunity.com/snapshots/",
+            ],
+        )

docs/deprecated/index.rst

@@ -14,8 +14,12 @@ objectives.
     :maxdepth: 1
 
     changed_keywords
+    dependency_links
     python_eggs
     easy_install
+    zip_safe
+    resource_extraction
     distutils/index
     distutils-legacy
     functionalities
+    running_commands

docs/deprecated/resource_extraction.rst

@@ -0,0 +1,54 @@
+.. _Automatic Resource Extraction:
+
+Automatic Resource Extraction
+=============================
+
+In a modern setup, Python packages are usually installed as directories,
+and all the files can be found on deterministic locations on the disk.
+This means that most of the tools expect package resources to be "real" files.
+
+There are a few occasions however that packages are loaded in a different way
+(e.g., from a zip file), which is incompatible with the assumptions mentioned above.
+Moreover, a package developer may also include non-extension native libraries or other files that
+C extensions may expect to be able to access.
+
+In these scenarios, the use of :mod:`importlib.resources` is recommended.
+
+Old implementations (prior to the advent of :mod:`importlib.resources`) and
+long-living projects, however, may still rely on the library ``pkg_resources``
+to access these files.
+
+If you have to support such systems, or want to provide backward compatibility
+for ``pkg_resources``, you may need to add an special configuration
+to ``setuptools`` when packaging a project.
+This can be done by listing as ``eager_resources`` (argument to ``setup()``
+in ``setup.py`` or field in ``setup.cfg``) all the files that need to be
+extracted together, whenever a C extension in the project is imported.
+
+This is especially important if your project includes shared libraries *other*
+than ``distutils``/``setuptools``-built C extensions, and those shared libraries use file
+extensions other than ``.dll``, ``.so``, or ``.dylib``, which are the
+extensions that setuptools 0.6a8 and higher automatically detects as shared
+libraries and adds to the ``native_libs.txt`` file for you.  Any shared
+libraries whose names do not end with one of those extensions should be listed
+as ``eager_resources``, because they need to be present in the filesystem when
+he C extensions that link to them are used.
+
+The ``pkg_resources`` runtime for compressed packages will automatically
+extract *all* C extensions and ``eager_resources`` at the same time, whenever
+*any* C extension or eager resource is requested via the ``resource_filename()``
+API.  (C extensions are imported using ``resource_filename()`` internally.)
+This ensures that C extensions will see all of the "real" files that they
+expect to see.
+
+Note also that you can list directory resource names in ``eager_resources`` as
+well, in which case the directory's contents (including subdirectories) will be
+extracted whenever any C extension or eager resource is requested.
+
+Please note that if you're not sure whether you need to use this argument, you
+don't!  It's really intended to support projects with lots of non-Python
+dependencies and as a last resort for crufty projects that can't otherwise
+handle being compressed.  If your package is pure Python, Python plus data
+files, or Python plus C, you really don't need this.  You've got to be using
+either C or an external program that needs "real" files in your project before
+there's any possibility of ``eager_resources`` being relevant to your project.

docs/deprecated/running_commands.rst

@@ -0,0 +1,23 @@
+Running ``setuptools`` commands
+===============================
+
+Historically, ``setuptools`` allowed running commands via a ``setup.py`` script
+at the root of a Python project, as indicated in the examples below::
+
+    python setup.py --help
+    python setup.py --help-commands
+    python setup.py --version
+    python setup.py sdist
+    python setup.py bdist_wheel
+
+You could also run commands in other circumstances:
+
+* ``setuptools`` projects without ``setup.py`` (e.g., ``setup.cfg``-only)::
+
+   python -c "import setuptools; setup()" --help
+
+* ``distutils`` projects (with a ``setup.py`` importing ``distutils``)::
+
+   python -c "import setuptools; with open('setup.py') as f: exec(compile(f.read(), 'setup.py', 'exec'))" develop
+
+That is, you can simply list the normal setup commands and options following the quoted part.

docs/deprecated/zip_safe.rst

@@ -0,0 +1,74 @@
+Understanding the ``zip_safe`` flag
+===================================
+
+The ``zip_safe`` flag is a ``setuptools`` configuration mainly associated
+with the ``egg`` distribution format
+(which got replaced in the ecosystem by the newer ``wheel`` format) and the
+``easy_install`` command (deprecated in ``setuptools`` v58.3.0).
+
+It is very unlikely that the values of ``zip_safe`` will affect modern
+deployments that use :pypi:`pip` for installing packages.
+Moreover, new users of ``setuptools`` should not attempt to create egg files
+using the deprecated ``build_egg`` command.
+Therefore, this flag is considered **obsolete**.
+
+This document, however, describes what was the historical motivation behind
+this flag, and how it was used.
+
+Historical Motivation
+---------------------
+
+For some use cases (such as bundling as part of a larger application), Python
+packages may be run directly from a zip file.
+Not all packages, however, are capable of running in compressed form, because
+they may expect to be able to access either source code or data files as
+normal operating system files.
+
+In the past, ``setuptools`` would install a project distributed
+as a zipfile or a directory (via the ``easy_install`` command or
+``python setup.py install``),
+the default choice being determined by the project's ``zip_safe`` flag.
+
+How the ``zip_safe`` flag was used?
+-----------------------------------
+
+To set this flag, a developer would pass a boolean value for the ``zip_safe`` argument to the
+``setup()`` function, or omit it.  When omitted, the ``bdist_egg``
+command would analyze the project's contents to see if it could detect any
+conditions that preventing the project from working in a zipfile.
+
+This was extremely conservative: ``bdist_egg`` would consider the
+project unsafe if it contained any C extensions or datafiles whatsoever.  This
+does *not* mean that the project couldn't or wouldn't work as a zipfile!  It just
+means that the ``bdist_egg`` authors were not yet comfortable asserting that
+the project *would* work.  If the project did not contain any C or data files, and did not
+attempt to perform ``__file__`` or ``__path__`` introspection or source code manipulation, then
+there was an extremely solid chance the project will work when installed as a
+zipfile.  (And if the project used ``pkg_resources`` for all its data file
+access, then C extensions and other data files shouldn't be a problem at all.
+See the :ref:`Accessing Data Files at Runtime` section for more information.)
+
+The developer could manually set ``zip_safe`` to ``True`` to perform tests,
+or to override the default behaviour (after checking all the warnings and
+understanding the implications), this would allow ``setuptools`` to install the
+project as a zip file. Alternatively, by setting ``zip_safe`` to ``False``,
+developers could force ``setuptools`` to always install the project as a
+directory.
+
+Modern ways of loading packages from zip files
+----------------------------------------------
+
+Currently, popular Python package installers (such as :pypi:`pip`) and package
+indexes (such as PyPI_) consider that distribution packages are always
+installed as a directory.
+It is however still possible to load packages from zip files added to
+:obj:`sys.path`, thanks to the :mod:`zipimport` module 
+and the :mod:`importlib` machinery provided by Python standard library.
+
+When working with modules loaded from a zip file, it is important to keep in
+mind that values of ``__file__`` and ``__path__`` might not work as expected.
+Please check the documentation for :mod:`importlib.resources`, if file
+locations are important for your use case.
+
+
+.. _PyPI: https://pypi.org

docs/index.rst

@@ -7,6 +7,10 @@ Documentation
 Setuptools is a fully-featured, actively-maintained, and stable library
 designed to facilitate packaging Python projects.
 
+It helps developers to easily share reusable code (in the form of a library)
+and programs (e.g., CLI/GUI tools implemented in Python), that can be installed
+with :pypi:`pip` and uploaded to `PyPI <http://pypi.org>`_.
+
 .. toctree::
    :maxdepth: 1
    :hidden: