Help people using Intersphinx against objects imported from a package __init__.py

[cblegare]

Subject: Intersphinx references to objects imported at package level can't be mapped.

Problem

References to objects that are imported in their package's __init__.py can't be referenced using the default objects inventory generation.

The issue here is inconsistent default behavior of sphinx-apidoc and intersphinx.

Procedure to reproduce the problem

Here is an example involving marshmallow

import marshmallow

class MySchema(marshmallow.Schema):
    pass

Build the apidoc then HTML docs using intersphinx:

sphinx.errors.SphinxWarning: /home/me/example.py:docstring of example.MySchema:1:py:class reference target not found: marshmallow.schema.Schema

No matter how I import the class, marshmallow.Schema is in objects.inv but intersphinx will always look for marshmallow.schema.Schema.

Work around

One could add aliased entries in the inventory in conf.py:

intersphinx_aliases = {
    # Alias for a class that was imported at its package level
    ('py:class', 'marshmallow.schema.Schema'):
        ('py:class', 'marshmallow.Schema'),
    # Alias for unavailable apidoc having a equivalent entry somewhere else 
    ('py:class', 'defusedxml.cElementTree'):
        ('py:class', 'xml.etree.ElementTree.ElementTree'),
}


def add_intersphinx_aliases_to_inv(app):
    from sphinx.ext.intersphinx import InventoryAdapter
    inventories = InventoryAdapter(app.builder.env)

    for alias, target in app.config.intersphinx_aliases.items():
        alias_domain, alias_name = alias
        target_domain, target_name = target
        try:
            found = inventories.main_inventory[target_domain][target_name]
            try:
                inventories.main_inventory[alias_domain][alias_name] = found
            except KeyError:
                continue
        except KeyError:
            continue


def setup(app):
    app.add_config_value('intersphinx_aliases', {}, 'env')
    app.connect('builder-inited', add_intersphinx_aliases_to_inv)

This workaround is IMO cleaner than using https://github.com/bskinn/sphobjinv to play with objects.inv files.

Varia

I don't know how things could be improved to help users get coherent results by default, but I do hope that this issue expose a behavior issue and this workaround help someone

[tk0miya]

I think this should be fixed on host side, not on referrer side. So I consider this is an issue of autodoc. The intersphinx extension allows to refer remote objects like they're defined in local. I know aliases can conceal the problem, but it is not perfect.

[cblegare]

I am not sure if relevant, but in both of the following cases:

import marshmallow.schema

class MySchema(marshmallow.Schema):
    pass

class MyOtherSchema(marshmallow.schema.Schema):
    pass

Intersphinx will always look for marshmallow.schema.Schema. Or is it an autodoc issue on the referer side?

[cblegare]

See also
https://stackoverflow.com/questions/22096187/how-to-make-sphinx-respect-importing-classes-into-package-with-init-py/24787344#24787344

[cblegare]

See also
https://stackoverflow.com/questions/30856279/how-to-use-sphinx-automodule-and-exposed-functions-in-init

[gaborbernat]

@tk0miya so what would be the solution for this? Adding built-in support for intersphinx_aliases, autogenerate it from autodoc?

The interesting code is at https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/autodoc/__init__.py#L785, here the real_modname is defined, the module that actually contains the object; while self.modname contains under what it's exposed. Perhaps the easiest fix is to use this to generate the aliases and within python domains resolve xref use that mapping to load what it maps too.

[cblegare]

@tk0miya If you would provide some preference on the solution, as asked by @gaborbernat , I might try and submit a PR

[tk0miya]

Oh, sorry. I've overlooked his comment.

My answer is the latter one, generating aliases by autodoc. This is not a problem only in intersphinx. As a first step, I added :canonical: option to the python domain (#7463). It allows to create an alias of the object. I thought the next step is generating :canonical: option in the autodoc. But I don't have time for it now. So it would be very helpful if you submit a PR :-)

[cblegare]

@tk0miya Thank you for this information

I drafted an implementation which broke a few (3) tests. These few edge cases definitely require a closer look. I'll submit the PR, probably this week, and work with your feedback.