sphinx.ext: incompatibility between autodoc and viewcode

[adamjstewart]

Describe the bug
Many projects use "import aliases" to simplify their code base. For example, in PyTorch, the Module class is defined in torch/nn/modules/modules.py, but can be imported as torch.nn.Module. The torch/nn/__init__.py file contains something like:

from torch.nn.modules.modules import Module

__all__ = ("Module",)

to make this possible. However, this creates two seemingly irreconcilable problems with autodoc and viewcode.

For autodoc, the documentation of these objects appears in torch.nn, but when creating links for these classes, autodoc wants to use Module.__module__, which points to torch.nn.modules.modules instead. Following the suggestion at https://stackoverflow.com/a/40040000/5828163, this can be "solved" by using:

Module.__module__ = "torch.nn"

However, this now introduces a problem with viewcode. Viewcode seems to also use the __module__ attribute to determine where the source code can be found, and links to this. If the __module__ attribute is overridden, now source code links no longer work.

Is this a known issue? Is there any workaround for this? I think an ideal scenario would be one where libraries don't have to override __module__ at all (viewcode now works) and autodoc becomes smart enough to understand that there are multiple possible locations where a class/function could be defined in the documentation. Note that this problem also affects intersphinx for the same reasons as autodoc.

To Reproduce
Steps to reproduce the behavior:

1. Use both autodoc and viewcode extensions in your conf.py
2. Use import aliases
3. Now links to classes/functions won't work properly
4. Override __module__ to fix this
5. Now links to source code won't work properly

Expected behavior
I would expect both of these extensions to be compatible in this common situation of import alias usage.

Your project
I'm working on this specifically in the context of PyTorch at pytorch/pytorch#61045 but it affects multiple projects.

Environment info

• OS: macOS 10.15.7
• Python version: 3.8.10
• Sphinx version: 4.3.1, 4.0.2, 3.2.0
• Sphinx extensions: sphinx.ext.autodoc, sphinx.ext.viewcode, sphinx.ext.intersphinx
• Extra tools: N/A

@ydcjeff @calebrob6

[adamjstewart]

@tk0miya do you have any thoughts on this?

[adamjstewart]

This has other unexpected side effects as well. autodoc_member_order = "bysource" does not work because Sphinx can't find the source code due to having to overwrite the __module__.

[tk0miya]

Sorry for the late response. Since 4.0, Sphinx supports "canonical" names for such "import aliased" objects. So __module__ hack has been no longer needed, AFAIK. Could you try this again without the hack, please?
And please let me know if it's not going well.

[adamjstewart]

@tk0miya I'm still seeing a lot of issues, see microsoft/torchgeo#317 for an example of this. Let me know if you aren't able to access https://readthedocs.org/projects/torchgeo/builds/15587403/. Basically, it's a bunch of errors like:

/Users/Adam/torchgeo/torchgeo/datasets/chesapeake.py:docstring of torchgeo.datasets.chesapeake.ChesapeakeCVPRDataModule.train_dataloader:: WARNING: py:class reference target not found: torch.utils.data.dataloader.DataLoader

because DataLoader is imported from and documented at torch.utils.data but is actually defined in torch.utils.data.dataloader. This is with Sphinx 4.3.1.

[tk0miya]

Oops... pytorch has still used Sphinx-3.5.4... To use the "canonical" names via intersphinx, remote project should also use the Sphinx v4.0+. Until then, you should use __module__ hack...
https://github.com/pytorch/pytorch/blob/master/docs/requirements.txt

[adamjstewart]

Let me see if I can get PyTorch to update to a newer Sphinx, thanks for the help!