NewType and TypeVar don't resolve correctly

[anselor]

Describe the bug
Autodoc generates warnings and doesn't resolve NewType or TypeVar correctly.

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/anselor/sphinx-issues-examples/
$ cd sphinx-issues-examples/typevar-namespace
$ python -m sphinx -b html docs/ doc-out/ -nvWT

Expected behavior
Documentation generated with TypeVar and NewType names that link to the documented declaration

Your project
https://github.com/anselor/sphinx-issues-examples/tree/main/typevar-namespace

Screenshots
If applicable, add screenshots to help explain your problem.

Environment info

• OS: Ubuntu 18.04, 20.04
• Python version: 3.6.9, 3.8.2
• Sphinx version: 3.2.1
• Sphinx extensions: sphinx.ext.autodoc
• Extra tools:

Additional context
Add any other context about the problem here.

• [e.g. URL or Ticket]

[anselor]

@tk0miya FYI - this is the new issue I created as you suggested.

[tk0miya]

@anselor I can't reproduce any warnings from your example.

This is a Dockerfile to reproduce the warning:

FROM python:3.8-slim

RUN apt update; apt install -y build-essential curl git make unzip vim
RUN git clone https://github.com/anselor/sphinx-issues-examples/
RUN pip install Sphinx
WORKDIR /sphinx-issues-examples/typevar-namespace/src
RUN pip install -e .
WORKDIR /sphinx-issues-examples/typevar-namespace/docs
RUN sphinx-build . _build/html

And its result:

Running Sphinx v3.3.1
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index

generating indices... genindex py-modindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in _build/html.

[anselor]

I was using 3.2.1 - just updated to sphinx 3.3.1 to make sure. The steps in your docker container do not match the same steps/options that I documented in the ticket.

I tried running it the way you have in your docker container and can confirm that sphinx doesn't report any errors but silently failed to resolve symbols or link them. Note that a locally declared TypeVar resolves correctly but a TypeVar imported from another module fails. I've underlined them in green and red in the output here:

For NewType it doesn't recognize the NewType name and generates this instead:

I would expect it to say:
special() -> SpecialType and link to the decleration of SpecialType

[tk0miya]

Sorry, I did not pass -n option to sphinx-build command. Now I reproduced the error. Okay, I'll take a look again.

FROM python:3.8-slim

RUN apt update; apt install -y build-essential curl git make unzip vim
RUN git clone https://github.com/anselor/sphinx-issues-examples/
RUN pip install Sphinx
WORKDIR /sphinx-issues-examples/typevar-namespace/src
RUN pip install -e .
WORKDIR /sphinx-issues-examples/typevar-namespace/docs
RUN sphinx-build -nvWT .  _build/html

Running Sphinx v3.3.1
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index


Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 361, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 535, in write
    self._write_serial(sorted(docnames))
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 545, in _write_serial
    self.write_doc(docname, doctree)
  File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 213, in pending_warnings
    memhandler.flushTo(logger)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 178, in flushTo
    logger.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1587, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1649, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 946, in handle
    rv = self.filter(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 807, in filter
    result = f.filter(record)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 421, in filter
    raise exc
sphinx.errors.SphinxWarning: /sphinx-issues-examples/typevar-namespace/src/issue/typevar/support.py:docstring of issue.typevar.support.MyClass.special::py:class reference target not found: NewType.<locals>.new_type

Warning, treated as error:
/sphinx-issues-examples/typevar-namespace/src/issue/typevar/support.py:docstring of issue.typevar.support.MyClass.special::py:class reference target not found: NewType.<locals>.new_type
The command '/bin/sh -c sphinx-build -nvWT  .  _build/html' returned a non-zero code: 2

[tk0miya]

I posted #8426 to fix the TypeVar warnings.

On the other hand, it is difficult to fix the case of NewType. It is hard to get the original name of the type in runtime. So please use autodoc_type_hints to resolve the name. It will resolve your case.
https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases

[anselor]

Can you not access the __name__ property?

https://github.com/python/cpython/blob/2837241f22be33a5597707b2aa723cb2cf6f3967/Lib/typing.py#L1942

[tk0miya]

Do you know how to detect is the type a kind of NewType? When should we access __name__?

[anselor]

They way it's implemented, NewType() returns a function that has 2 additional attributes: __name__ and __supertype__.
In my code that does some runtime type validation I check for these things:

Does __qualname__ equal NewType.<locals>.new_type ?
Do __name__ and __supertype__ exist as attributes?

[tk0miya]

Hmm... I feel it's a bit fragile. But, I understand it is better to support NewType as possible by default. Additionally, I need to know how to get the module of the new type. In this case, we need to get issue.typevar.support.SpecialInt from the NewType object. I'll take a look later.

[anselor]

Hmm, I see what you mean. It looks like the __name__ field is from the string passed into the typing.NewType() function. The original module/namespace context doesn't appear to be preserved in any way.

Something probably needs to be submitted to the Python project if we want that changed.

Would it be possible to have the option to automatically register any NewType declarations in the autodoc_type_aliases dictionary?
Also, I noticed in the generated output that NewType and TypeVar don't have any automatic indication of what type they're declared as. Perhaps that's something that can be resolved in the context of where it is declared? For example, it would be nice if TypeVar auto-documented what, if any, type it is bound to and whether it is invariant, covariant, or contravariant (https://github.com/python/cpython/blob/2837241f22be33a5597707b2aa723cb2cf6f3967/Lib/typing.py#L567).
And for NewType, it would be nice to auto-document the __supertype__. Here is the current generated output for TypeVar and NewType:

If the above suggestions don't work, a reasonable fallback approach would be to acknowledge that it is a NewType alias and link to the provided __supertype__.

[tk0miya]

Would it be possible to have the option to automatically register any NewType declarations in the autodoc_type_aliases dictionary?

That's strange. autodoc_type_aliases is a type hint for the types that Sphinx can't resolve. It is not needed if we can get the correct information of the NewType. To fill the configuration, we need to know the module of it.

Also, I noticed in the generated output that NewType and TypeVar don't have any automatic indication of what type they're declared as.

Good point. +1 for your idea. Could you file another issue, please? Then somebody would work for it in the future (including me!).

[anselor]

I've been unable to try the latest changes because there are regressions on 3.4 that fail to run for me.

[tk0miya]

I just merge #8426. It will be released as 4.0 this April.
Thanks!

[anselor]

@tk0miya I tried various ways to use autodoc_type_aliases and it doesn't seem to do anything useful. The only way I'm able to move forward is adding them all as nitpick ignore.

[tk0miya]

If you're saying about NewType, there is nothing to do from Sphinx. Please request python to store the location to NewType objects to document it.

[anselor]

I thought you'd suggested that using autodoc_type_aliases to manually map the name over would be a work-around? When I added the NewType names to the map it had no effect on the output.

[anselor]

I figured out a work-around for making NewType resolve. @tk0miya

After a NewType declaration, If I add a line afterwards to manually specify the __qualname__ Sphinx seems to figure things out.

SpecialInt = NewType('SpecialInt', int)
SpecialInt.__qualname__ = f'{__name__}.{SpecialInt.__name__}'

[anselor]

@tk0miya
Would it be possible to, on startup, have Sphinx monkeypatch NewType. Then, implement a NewType wrapper that inspects the call stack to figure out the context from which it was called. Using that, fill out qualname before returning it?