sphinx.ext.autosummary with automatic recursion doesn't follow legitimate import statement in __init__.py file

[JamesALeedham]

Describe the bug
Sphinx fails to import & document a module that is itself imported in an __init__.py file. I'm at a loss as to understand why; on the face of it, there's nothing wrong with this import statement. If I comment out the import statement, Sphinx imports & documents the module perfectly. Why is this?

To Reproduce

1. Clone git@github.com:JamesALeedham/lusid-python-tools.git and checkout the sphinx-bug-query branch.
2. View the copy of the docs hosted here and observe that the lusidtools.logger.LusidLogger class is not documented (ie. page is empty).
3. In the repo, edit lusidtools/logger/__init__.py and comment out the single import statement, ie. # from lusidtools.logger.LusidLogger import LusidLogger.
4. Follow the instructions in the docs Readme to rebuild and view the docs locally as HTML.

With the import statement in __init__.py commented out, Sphinx documents the lusidtools.logger.LusidLogger class perfectly. (Here's an example of what this page then looks like).

Environment info

• Sphinx version: 3.5.3
• Sphinx extensions: sphinx.ext.autosummary with the brilliant :recursive: option

[Chilipp]

hey everyone! I found the problematic code:

sphinx/sphinx/ext/autosummary/__init__.py

Lines 633 to 650 in 952866c

	for j in reversed(range(1, len(name_parts) + 1)):
	last_j = j
	modname = '.'.join(name_parts[:j])
	try:
	import_module(modname)
	except ImportError:
	continue
	if modname in sys.modules:
	break
	if last_j < len(name_parts):
	parent = None
	obj = sys.modules[modname]
	for obj_name in name_parts[last_j:]:
	parent = obj
	obj = getattr(obj, obj_name)
	return obj, parent, modname

it's totally non-trivial I'd say. I try to explain it with a minimal example that you can access here https://github.com/Chilipp/autosummary-issue, or as zip file here. Assume your code looks like this:

autosummary_issue/
├── __init__.py
├── issue.py

with

• __init__.py

   from . import issue

• issue.py

  import autosummary_issue
  
  class DummyClass:
      """Some dummy class.""" 

• index.rst

  .. automodule:: autosummary_issue.issue
      :members:
  
  .. autosummary::
  
      autosummary_issue.issue.DummyClass

when the import_by_name function get's called, it is getting two prefixes: "autosummary_issue.issue" from the .. currentmodule:: directive that is implicitly added by .. automodule::, and None.

The loop now goes through the prefixes and starts with creating the prefixed_name = "autosummary_issue.issue.autosummary_issue.issue.DummyClass". This is obviously the wrong name, but the problem is in this line:

sphinx/sphinx/ext/autosummary/__init__.py

Line 649 in 952866c

obj = getattr(obj, obj_name)

Here, the _import_by_name function should break, but it it doesn't, "autosummary_issue.issue.autosummary_issue.issue.DummyClass" is actually accessible because of the import statement (from . import issue) in __init__.py. So autosummary thinks that autosummary_issue.issue.autosummary_issue.issue.DummyClass is a valid object, tries to look it up in the index and obviously fails, because the index know that the real name is autosummary_issue.issue.DummyClass.

[JamesALeedham]

Hey @Chilipp , thanks for looking into this and glad you managed to find something...

One of the things that's puzzling me is that I ran Sphinx over a sister repo that also has import statements in __init__.py files and it worked fine. To be clear, it was exactly the same Sphinx setup (sphinx.ext.autosummary with :recursive: option) in exactly the same environment, and it documented the package perfectly (ie. I didn't have to comment out any of the import statements in the __init__.py file linked above to get Sphinx to work).

So perhaps a better title for this issue would be: why does Sphinx fail to follow some legitimate import statements in __init__.py files..?

[Chilipp]

Hey @JamesALeedham ! Did you use the same rst file in both cases? Because the critical thing in my example from above is the combination

1. The import
2. the .. currentmodule:: directive that is implicitly added with .. automodule::.

If you drop the automodule thing, or put the automodule behind the autosummary, everything works well

[JamesALeedham]

Just to be clear about my setup, I'm not handcrafting the .rst, I'm relying on :recursive: to do that for me: https://github.com/JamesALeedham/lusid-python-tools/blob/sphinx-setup/docs/api.rst

I guess Autosummary is then using my custom module template and class template to do the actual doc generation...

(At this point, I've probably reached my limit in understanding how Sphinx works... 😞 )

But yes, the Sphinx setup is 100% identical for both repos - one in which import statements in __init__.py files cause a problem, and one in which they don't.

[DeltaRazero]

Just to be clear about my setup, I'm not handcrafting the .rst, I'm relying on :recursive: to do that for me: https://github.com/JamesALeedham/lusid-python-tools/blob/sphinx-setup/docs/api.rst

I guess Autosummary is then using my custom module template and class template to do the actual doc generation...

(At this point, I've probably reached my limit in understanding how Sphinx works... disappointed )

But yes, the Sphinx setup is 100% identical for both repos - one in which import statements in __init__.py files cause a problem, and one in which they don't.

Having this problem as well.

[MCRE-BE]

#8963
is linked

[MCRE-BE]

Hey @JamesALeedham ! Did you use the same rst file in both cases? Because the critical thing in my example from above is the combination

1. The import
2. the .. currentmodule:: directive that is implicitly added with .. automodule::.

If you drop the automodule thing, or put the automodule behind the autosummary, everything works well

Switching automodule after autosummary manually worked for me. With the option "autosummary_generate_overwrite = False" allows to have something workable (more or less) and manage only the module level manually.

[domonik]

Not sure whether I got the topic correctly. I was trying to get functions imported in the __init__.py documented also at the corresponding module level.
I just played around a bit and changing a single line would at least create an additional entry in the module level for such functions and classes.

sphinx/sphinx/ext/autosummary/generate.py

Line 254 in 952866c

if imported or getattr(value, '__module__', None) == obj.__name__:

changing this to

        if imported or obj.__name__ in getattr(value, '__module__', None):

will work.
However, i have looked into the code the first time. So no idea wehther I missed something important and this would damage other parts of the code.

[Naggafin]

Is there a workaround for this in the meanwhile?

[Naggafin]

For anyone dealing with this issue as well, I found a solution that worked for me. My auto generated rst looked like this:

..
  module.rst

mypackage.mymodule
==================

.. automodule:: mypackage.mymodule
        
            .. rubric:: Functions

            .. autosummary::
                :toctree:
                :nosignatures:
                :template: autosummary/base.rst
                
                    some_function
                    another_function

    
        .. rubric:: Modules

        .. autosummary::
            :toctree:
            :template: autosummary/module.rst
            :recursive:
            
                mypackage.mymodule.mysubmodule

The problem, as @Chilipp discovered, lies with automodule. Removing that without changing anything though will treat the functions as modules in and of themselves though. I ended up having to transform my rst file to look like this to get it working:

..
  module.rst

mypackage.mymodule
==================

.. rubric:: Functions

.. autosummary::
    :toctree:
    :nosignatures:
    :template: autosummary/base.rst
    
        mypackage.mymodule.some_function
        mypackage.mymodule.another_function


.. rubric:: Modules

.. autosummary::
    :toctree:
    :template: autosummary/module.rst
    :recursive:
    
        mypackage.mymodule.mysubmodule

Notice that the functions are given fully qualified import paths. Hope this helps someone else!