By default, make Autosummary templates generate documentation pages for classes, functions, exceptions etc

[JamesALeedham]

Is your feature request related to a problem? Please describe.
The new :recursive: option for sphinx.ext.autosummary is great. I can point Sphinx at the top of my package and have it find every module, however deeply nested. For each module it creates neat summary tables listing all the attributes, classes, exceptions and functions in those modules. This is all fantastic.

To generate actual documentation pages for these attributes, classes, exceptions and functions, however, and link to them from the summary tables, I have to copy the default module.rst and class.rst templates locally and edit them. I would like to think this is what most people would want. So I think it should be the default behavior - and those who don't want this should have to customize the templates.

Describe the solution you'd like
I've described the solution I'd like to see here. There's a Github test implementation here and a sample ReadTheDocs user experience here.

(I'm sure you know what the current user experience is like but, just to be clear, you can reproduce it by cloning the Github test project and deleting the :template: custom-module-template.rst line in docs/source/index.rst.)

Basically, the solution involves adding a :toctree: option to every .. autosummary:: directive in the default module.rst template so that documentation pages get generated for attributes, classes, exceptions and functions out-of-the-box, like this:

site-packages/sphinx/ext/autosummary/templates/autosummary/module.rst (additional lines noted on right):

{{ fullname | escape | underline}}

.. automodule:: {{ fullname }}
  
   {% block attributes %}
   {% if attributes %}
   .. rubric:: Module Attributes

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in attributes %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block functions %}
   {% if functions %}
   .. rubric:: {{ _('Functions') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in functions %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block classes %}
   {% if classes %}
   .. rubric:: {{ _('Classes') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in classes %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block exceptions %}
   {% if exceptions %}
   .. rubric:: {{ _('Exceptions') }}

   .. autosummary::
      :toctree:                                          <-- add this line
   {% for item in exceptions %}
      {{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

{% block modules %}
{% if modules %}
.. rubric:: Modules

.. autosummary::
   :toctree:
   :recursive:
{% for item in modules %}
   {{ item }}
{%- endfor %}
{% endif %}
{% endblock %}

[tk0miya]

Thank you for proposal. Good idea!
I think it would be better to generate javadoc-like documents. So +1 for adding :toctree: option to module.rst.
https://docs.oracle.com/en/java/javase/13/docs/api/java.logging/java/util/logging/package-summary.html

On the other hand, I prefer to show the methods and attributes of the class in the same file (like Javadoc does). So it would be nice if we'll add autoclass or automethod (autoattributes) to class.rst.
https://docs.oracle.com/en/java/javase/13/docs/api/java.logging/java/util/logging/Logger.html

Anyway, the modification of the template brings a breaking change. So we have to change them on 4.0 release.

[tk0miya]

Note: Adding :toctree: option to module.rst causes recursive stub-file generation for the given module. We should consider the side effect of the change carefully.

[rickstaa]

@JamesALeedham First of all, thanks a lot for the solution on the StackOverflow question. I think it would make the auto_summary extension even more useful. There are, however, several problems I noticed while using the the StackOverflow question or the solution in this issue:

Stackoverflow version

The solution breaks if classes are imported onto a higher namespace using an init.py file

Let's say we have the following project structure:

└── package
    └── submodule
        ├── __init__.py
        └── subsubmodule
            └── subsubsubmodule.py

Now try to make the package.submodule.subsubmodule.subsubsubmodule.Class available on the package.submodule namespace using the following __init__.py file:

package.submodule.init.py

from package.submodule.subsubmodule.subsubmodule import Class

This prevents the auto summaries for the classes, functions and methods of the package.submodule.subsubmodule.subsubsubmodule from being generated and showing up using the :recursive: option. I still have to add the following code to make them show up manually:

   .. rubric:: Classes

   .. autosummary::
      :toctree:
      :template: custom-class-template.rst
   
      Class


   .. rubric:: Functions

   .. autosummary::
      :toctree:
   
      function_1

This is related to this sphinx issue. This can be solved by using the autosummary_imported_members = True option so should not be such a problem. I however think it would be more user friendly if the autosummary extension was able to show Classes and functions on their original namespace when they are imported onto another namespace inside the same package. This would lead to a cleaner API and also shows docstrings that are placed at the top of the imported modules.

The templates don't work nicely with intersphinx bindings

When I use this workaround combined with the intersphinx bindings, it could lead to possible errors if a class inherits from a base class that contains sphinx errors. I, for example, use the https://github.com/mr-ubik/tensorflow-intersphinx/ TensorFlow bindings, which contain several errors. Therefore, maybe care should be taken when hardcoding the :inherited-members: flag in the template as it seems to overwrite the autodoc_inherit_docstrings = False setting. Just something we should consider when implementing the feature. I think this is related to the comment of @tk0miya.

The version in this issue

When I omit the custom-class-template.rst the problems go away. Now, however, classes and functions are not documented.

[JamesALeedham]

Hi @rickstaa , thanks for getting back to me. I admit I only tested my solution on a few live projects, and didn't set out to test it exhaustively on a wide set of possible scenarios, so not surprised you've found cases where it doesn't work! Thanks for trying it out...

I'll be guided by what you experts think is best, of course. For my part, when getting started with Sphinx I found it hard to get something that 'quickly' & 'nearly' worked so that I could see I was on the right tracks (and from the responses I've had on StackOverflow and other places I am not alone!).

The goal with my solution (or something similar) is to get a majority of people up and running with something tangible out-of-the-box a bit quicker, and then let people explore Sphinx's myriad settings and options to tailor their experience at their own leisure. I hope it is a step in the right direction there, though naturally I realise it won't entirely work for everyone...

[rickstaa]

@JamesALeedham, I think your solution is good as it works in most use cases. I use it in most of my projects. My previous comment was mainly to provide the sphinx developers like @guyer and @tk0miya with the uses cases in which it does not work. I'm not familiar enough with the sphinx and sphinx.ext.autosummary codebase to develop a solution that also works in the aforementioned use cases.

[guyer]

@rickstaa I'm not a sphinx developer!

[rickstaa]

Ha @guyer sorry for making that assumption.

[dan0nchik]

Hey guys! Any progress on this feature?

[vineetsharma883]

Hey there, waiting for this very useful feature. Can someone share when it would be released in Sphinx?

[vineetsharma883]

I am working with it where I have a following folder structure:-

.
├── README.md
├── __init__.py
├── docs
│   ├── Makefile
│   ├── make.bat
│   └── source
│       ├── _autosummary
│       │   ├── ...
│       ├── _templates
│       │   ├── custom-class-template.rst
│       │   └── custom-module-template.rst
│       ├── conf.py
│       └── index.rst
├── extract
│   ├── README.md
│   ├── __init__.py
│   ├── folder_1
│   │   ├── a.py
│   │   ├── __init__.py
│   │   ├── b.py
│   │   ├── c.py
│   ├── folders_2
│   │   ├── __init__.py
│   │   ├── d.py
│   │   └── e.py
│   ├── folder_3
│   │   ├── __init__.py
│   │   ├── f.py
└── transform
    ├── __init__.py
    ├── g.py
    └── h.py

The script for templates is completely same as given on stackoverflow. Following is my code for index.rst and config.py:-
index.rst

Welcome to sample documentation!
======================================
Docs - Sample repository

.. autosummary::
   :toctree: _autosummary
   :template: custom-module-template.rst
   :recursive:

   extract
   transform

config.py

import os
import sys
sys.path.insert(0, os.path.abspath('../..'))

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.autosummary',
    'sphinx.ext.coverage',
    'sphinx.ext.napoleon',
    'sphinx.ext.viewcode'
]

templates_path = ['_templates']
exclude_patterns = ['_build', '_templates']

html_theme = 'sphinx_rtd_theme'
html_static_path = ['_static']
autosummary_generate = True

However, with this code I am unable to access the documentation in separate page for classes and functions in a.py, b.py, c.py, d.py, e.py, f.py, g.py, h.py until I have following code in index.rst:-

Welcome to titan documentation!
======================================
Docs - Titan repository

.. autosummary::
   :toctree: _autosummary
   :template: custom-module-template.rst
   :recursive:

   extract.folder_1.a
   extract.folder_1.b
   extract.folder_1.c
   transform.g
   transform.h
   ....
   ....
   ....

i.e. specifying each file individually.

• Demo
• Sample folder structure with versions on dependencies in requirements.txt

[buhrmann]

Hi, I'm trying to use the recursive option with the modified template to create stubs for functions, attributes etc. But, this also includes attributes, functions etc. in the navigation toctree. Is there a way to recursively generate all stubs (with corresponding links in the summary tables), but limit the depth of the navigation (or limit it to entries only for packages and modules)?

[jpfeuffer]

@buhrmann Did you find a solution? I am having the same problem.

[luis-chaves-visa]

Thank you so much @JamesALeedham, being new to autosummary I completely expected what you suggest to be the expected behaviour, this was driving me mad!! 🍔

[fepegar]

Thanks for sharing, @mhostetter. Your docs look amazing!

[NikosAlexandris]

I feel the need to share with my fellow Sphinx users an amazing new theme that's solved this issue for me -- Sphinx Immaterial. It has a feature python-apigen that is a modified version of autosummary. It will recursively document an entire package/module, or several.

Instead of producing an autosummary table, it produces a list of signature summaries which are hyperlinked to the individual object documentation pages. Here is an example API reference from my open-source project using Sphinx Immaterial.

In addition to recursive autosummary/autodoc, it monkey patches many bug fixes into Sphinx and adds many new features. The development is very active. I would recommend you all look into it. If you have issues, open an issue or discussion on https://github.com/jbms/sphinx-immaterial and we'll help you use it. 👍

Love the theming in https://galois.readthedocs.io/en/stable/api/!

[siddharth-krishna]

Another alternative is to use sphinxcontrib-apidoc. I prefer this to sphinx-immaterial because it does not require one to group classes/functions, and automatically generates a hierarchical documentation for all files/modules in a package.