Add get_members function to apidoc templates

The `module.rst_t` and `package.rst_t` templates used by apidoc now have
a `get_members` function that allows to obtain the member of the current
module or package in various forms and filtered by "type" (exception,
class, function, data).

This provides a powerful mechanism for writing more advanced templates.
Specifically, it allows to generate structured summaries of the contents
of a module, preceding the detailed `automodule` references.
See https://krotov.readthedocs.io/en/stable/API/krotov.html for an
example.

Extensive documentation of all the templating capabilities is included
in the apidoc man page in the documentation.

This functionality is a backport of the "better-apidoc" package
(https://github.com/goerz/better-apidoc).

AUTHORS

@@ -32,6 +32,7 @@ Other contributors, listed alphabetically, are:
 * Josip Dzolonga -- coverage builder
 * Buck Evan -- dummy builder
 * Matthew Fernandez -- todo extension fix
+* Michael Goerz -- apidoc templating
 * Hernan Grecco -- search improvements
 * Horst Gutmann -- internationalization support
 * Martin Hans -- autodoc improvements

CHANGES

@@ -51,6 +51,7 @@ Features added
 * #6707: C++, support bit-fields.
 * #267: html: Eliminate prompt characters of doctest block from copyable text
 * #6729: html theme: agogo theme now supports ``rightsidebar`` option
+* #6768: apidoc: The templates used by apidoc can now use a ``get_members`` function
 
 Bugs fixed
 ----------

doc/man/sphinx-apidoc.rst

@@ -107,7 +107,7 @@ These options are used when :option:`--full` is specified:
 
 .. option:: -a
 
-   Append module_path to sys.path.
+   Append *MODULE_PATH* to ``sys.path``.
 
 .. option:: -H <project>
 
@@ -126,29 +126,260 @@ These options are used when :option:`--full` is specified:
 
    Sets the project release to put in generated files (see :confval:`release`).
 
-.. rubric:: Project templating
-
-.. versionadded:: 2.2
-   Project templating options for sphinx-apidoc
 
 .. option:: -t, --templatedir=TEMPLATEDIR
 
    Template directory for template files.  You can modify the templates of
-   sphinx project files generated by apidoc.  Following Jinja2 template
+   sphinx project files generated by apidoc.  The following Jinja2 template
    files are allowed:
 
    * ``module.rst_t``
    * ``package.rst_t``
    * ``toc.rst_t``
+
+   See the section :ref:`apidoctemplating` below for details.
+
+   In addition, when :option:`--full` is specified,
+   :program:`sphinx-quickstart` allows for the following templates:
+
    * ``master_doc.rst_t``
    * ``conf.py_t``
    * ``Makefile_t``
    * ``Makefile.new_t``
    * ``make.bat_t``
    * ``make.bat.new_t``
 
-   In detail, please refer the system template files Sphinx provides.
-   (``sphinx/templates/apidoc`` and ``sphinx/templates/quickstart``)
+   Please refer the to system template files in ``sphinx/templates/quickstart``
+   for details.
+
+   .. versionadded:: 2.2
+
+
+.. _apidoctemplating:
+
+Templating
+----------
+
+.. versionadded:: 2.2
+   Project templating options for sphinx-apidoc
+
+The TOC template
+~~~~~~~~~~~~~~~~
+
+The template for the TOC file, which is generated unless :option:`--no-toc` is
+given, is in the file ``toc.rst_t``. It uses the following Jinja2 variables:
+
+.. data:: header
+
+   The :confval:`project` name, cf. :option:`-H`
+
+.. data:: maxdepth
+
+   Maximum depth for the generated table of contents file, see :option:`-d`
+
+.. data:: docnames
+
+   A sorted list of the modules in the TOC
+
+The default ``toc.rst_t`` is
+
+.. literalinclude:: ../../sphinx/templates/apidoc/toc.rst_t
+   :language: jinja
+
+The package template
+~~~~~~~~~~~~~~~~~~~~
+
+The package template (``package.rst_t``) is used to render packages or implicit
+name spaces. It uses the following Jinja2 variables:
+
+.. data:: pkgname
+
+The fully qualified package name
+
+.. data:: subpackages
+
+List of fully qualified sub-package names (if any)
+
+.. data:: submodules
+
+List of fully qualified sub-module names (if any)
+
+.. data:: is_namespace
+
+Whether or not the template is used to render an implicit name space
+
+.. data:: modulefirst
+
+Whether :option:`--module-first` was given
+
+.. data:: separatemodules
+
+Whether :option:`--separate` was given
+
+.. data:: automodule_options
+
+A comma-separated list of ``automodule`` directives, see
+:envvar:`SPHINX_APIDOC_OPTIONS`.
+
+.. data:: show_headings
+
+True unless :option:`--no-headings` was given
+
+
+The default ``package.rst_t`` is
+
+.. literalinclude:: ../../sphinx/templates/apidoc/package.rst_t
+   :language: jinja
+
+
+The module template
+~~~~~~~~~~~~~~~~~~~
+
+The module template is used to render modules. This happens only if
+:option:`--separate` was given, or if *MODULE_PATH* contains standalone
+modules instead of packages.
+
+It uses the :data:`automodule_options` and :data:`show_headings` variables of
+``package.rst_t``, and additionally:
+
+.. data:: qualname
+
+The fully qualified name of the module.
+
+.. data:: basename
+
+An alias for :data:`qualname`.
+
+The default ``module.rst_t`` is
+
+.. literalinclude:: ../../sphinx/templates/apidoc/module.rst_t
+   :language: jinja
+
+
+Accessing module members in templates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 2.3
+   Advanced templating with ``get_members``
+
+Both the ``package.rst_t`` and ``module.rst_t`` templates have access to a
+:func:`get_members` function that allows to extract the members of the package or
+module being rendered:
+
+
+.. py:function:: get_members(fullname, typ=None, include_imported=False, out_format='names', in_list=None, include_private=*private*, known_refs=None)
+
+   Return a list of members.
+
+   :param str fullname:
+      The fully qualified name of the module for which to get the members
+
+   :param Optional[str] typ:
+      One of None, 'function', 'class', 'exception', 'data'. If None, return
+      members of any type. Otherwise return only members of the given `typ`
+
+   :param bool include_imported:
+      If True, include members that are imported from other modules. If False,
+      only return members that are defined directly in the module
+
+   :param str out_format:
+      One of 'names', 'fullnames', 'refs', and 'table'
+
+   :param Optional[Union[str,Tuple[str]] in_list:
+      If not None, name or tuple of names of module attribute(s) (e.g.
+      ``'__all__'`` or ``('__all__', '__private__')``). Only members whose
+      names appears in the list(s) will be returned.
+
+   :param bool include_private:
+      If True, include members whose names starts with an underscore. Defaults
+      to False unless :option:`--private` is given.
+
+   :param Optional[Union[dict,str]] known_refs:
+      If not None, a mapping of names to rull rst-formatted references. If
+      given as a str, the mapping will be taken from the module attribute of
+      the given name. This is used only in conjunction with
+      ``out_format=refs``, to override automatically detected reference
+      location, or to provide references for object that cannot be located
+      automatically. The most common example for this is data members that are
+      exported by a package but are defined in a sub-module.
+
+   :return:
+      a list of strings, depending on `out_format`.
+
+      * If ``out_format='names'`` (default), the simple names of all qualifying
+        members.
+
+      * If ``out_format='fullnames'``, the fully qualified names of all
+        qualifying members.
+
+      * If ``out_format='refs'``, rst-formatted links for all qualifying
+        members. The links use an appropriate role (``:class:``, ``:func:``
+        etc.) depending on the type of the member, and point to the original
+        location defining that member (which may be in a sub-module).  The
+        `known_refs` dictionary allows to override this.
+
+      * If ``out_format='table'``, a list of lines for a rst table similar to
+        that generated by the autosummary plugin (left column is linked member
+        names, right column is first sentence of the docstring).
+
+   :rtype: List[str]
+
+Note that for data members, it is not always possible to determine whether they
+are imported or defined locally. In this case, `in_list` and `known_refs` may
+be used to achieve the desired result.
+
+If using ``in_list='__all__'`` for a package you may also have to use
+``include_imported=True`` to get the full list (as packages typically export
+members imported from their sub-modules)
+
+The :func:`get_members` function is easiest to use inside a template by using
+Jinja2's ``set`` command::
+
+   {%- set members = get_members() %}
+
+After that, the variable `members` is available in the template.
+
+The ability to filter members by type allows to write templates that render a
+detailed and structured summary of a module, see the example below.
+
+.. warning::
+
+   When rendering a template that uses :func:`get_members`, the package/module
+   must be importable! Thus, it may be required to pass the :option:`-a` option.
+
+
+
+Example for templates using ``get_members``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The files below provide a full example of an advanced set of templates. These
+are intended for documenting a package containing sub-modules (and potentially
+sub-packages). They require that ``apidoc`` is called with :option:`--separate`.
+
+For each package and module, the templates render a "Summary" section first.
+This section contain a tabular ``autosummary`` of all members, grouped by category
+(exceptions, classes, functions, data). It also documents the ``__all__`` list,
+linking every member of that list to its original definition. For data members
+which are imported and exposed from a sub-module, links are read from the
+``__known_refs__`` attribute of the module.
+
+The "Summary" section is followed by a "Reference" section that contains the
+normal ``automodule`` documentation for all members of the module.
+
+.. warning::
+
+   These templates require that the package is importable when ``apidoc`` is
+   called. Also, they rely on the ``automodule`` extension, which must be
+   enabled in ``conf.py``.
+
+
+.. literalinclude:: templates_advanced/module.rst_t
+   :language: jinja
+   :caption: ``module.rst_t``
+
+.. literalinclude:: templates_advanced/package.rst_t
+   :language: jinja
+   :caption: ``package.rst_t``
 
 Environment
 -----------

doc/man/templates_advanced/README.md

@@ -0,0 +1,6 @@
+These template file serve as an example of how to use the `get_members` function.
+
+They are included in the `apidoc` man page.
+
+A slightly expanded version of these templates is part of the [automated tests
+for `apidoc` with templates](../../../tests/roots/test-apidoc-templates/)

doc/man/templates_advanced/module.rst_t

@@ -0,0 +1,83 @@
+{%- set members = get_members() %}
+{%- set all = get_members(in_list='__all__', include_imported=True) %}
+{%- set all_refs = get_members(in_list='__all__', include_imported=True, out_format='refs', known_refs='__known_refs__') -%}
+{%- set exceptions = get_members(typ='exception') -%}
+{%- set classes = get_members(typ='class') -%}
+{%- set functions = get_members(typ='function') -%}
+{%- set data = get_members(typ='data', in_list='__all__') -%}
+
+{{ [qualname, "module"] | join(" ") | e | heading }}
+
+.. currentmodule:: {{ qualname }}
+
+.. automodule:: {{ qualname }}
+{%-   if members %}
+    :members: {{ members|join(", ") }}
+    :undoc-members:
+    :show-inheritance:
+{%    endif -%}
+
+{%-   if members or all %}{# summary_members_or_all #}
+
+    Summary
+    -------
+
+{%-     if exceptions %}
+
+    Exceptions:
+
+    .. autosummary::
+        :nosignatures:
+{%        for item in exceptions %}
+        {{ item }}
+{%-       endfor %}
+{%-     endif %}
+
+{%-     if classes %}
+
+    Classes:
+
+    .. autosummary::
+        :nosignatures:
+{%        for item in classes %}
+        {{ item }}
+{%-       endfor %}
+{%-     endif %}
+
+{%-     if functions %}
+
+    Functions:
+
+    .. autosummary::
+        :nosignatures:
+{%        for item in functions %}
+        {{ item }}
+{%-       endfor %}
+{%-     endif %}
+
+{%-     if data %}
+
+    Data:
+
+    .. autosummary::
+        :nosignatures:
+{%        for item in data %}
+        {{ item }}
+{%-       endfor %}
+{%-     endif %}
+
+{%-     if all_refs %}
+
+    ``__all__``: {{ all_refs|join(", ") }}
+{%-     endif %}
+
+
+{%-   endif %}{# summary_members_or_all #}
+
+
+{%-   if members %}
+
+    Reference
+    ---------
+
+{%    endif %}