Fix Numba decorated function docstrings #173

thisac · 2020-05-21T01:43:01Z

Context:
The generated docstrings for functions/methods decorated with Numba's @jit decorator didn't render properly since:

The decorator obscured the signature to Sphinx.
numba and numba.jit were mocked out inconf.py causing magicmock to provide its docstring instead of numba (which actually provides the correct, original docstring).

Description of the Change:

An autodoc-signature-process function was added to conf.py that checks whether the function is a Numba object, and in that case retrieves the original signature and forwards it to Sphinx.
The mocked out functions/modules numba and numba.jit are removed from conf.py, letting the correct docstrings be rendered.
The mocked out functions/modules numpy and numpy.dtype are removed from conf.py, letting the correct dtype be rendered inside the signatures.

Also fixed a few docstrings that weren't rendered correctly due to missing line-breaks.

Benefits:
The docstrings and signatures now render as they should.

Possible Drawbacks:
I'm not sure exactly why numpy and numba were mocked out in conf.py (nor the other funtions/method), so I might have broken something on the way. The documentation rendered nicely locally though.

Related GitHub Issues:
N/A

codecov · 2020-05-21T02:03:11Z

Codecov Report

Merging #173 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #173   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           14        14           
  Lines         1002      1002           
=========================================
  Hits          1002      1002

Impacted Files	Coverage Δ
thewalrus/csamples.py	`100.00% <ø> (ø)`
thewalrus/fock_gradients.py	`100.00% <ø> (ø)`
thewalrus/samples.py	`100.00% <ø> (ø)`
thewalrus/symplectic.py	`100.00% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf7a19e...8d203f7. Read the comment docs.

josh146

This is awesome, thanks for fixing this @thisac!

Since this requires that Numba not be mocked out, one thing we'll have to double check is if Numba is installable on readthedocs. I've created a hidden build of this branch here: https://hafnian.readthedocs.io/en/numba_fix_decorator, and we can track builds here: https://readthedocs.org/projects/hafnian/builds/

josh146 · 2020-05-21T02:40:40Z

docs/conf.py

 MOCK_MODULES = [
-    'numpy',
-    'numpy.dtype',
+    # 'numpy',
+    # 'numpy.dtype',
    'scipy',
    'scipy.special',
    'scipy.optimize',
    'scipy.stats',
    'cython',
    'thewalrus.libwalrus',
-    'numba',
-    'numba.jit'
+    # 'numba',
+    # 'numba.jit'
    ]


There were originally several reasons why these modules are mocked out:

There is a bug in Sphinx (it might be fixed now) where Sphinx-autodoc would mistakenly include NumPy function documentation in projects that simply import it. This happened in Strawberry Fields, hence why it is probably mocked out here. I think it's okay to un-mock it now.

Mocking allows Sphinx to import the project and inspect docstrings without requiring the dependencies be installed. This is quite important for 'thewalrus.libwalrus', as we don't want to have to build the C++ code on the RTDs environment.

I've noticed that scipy and numpy are now in the doc/requirements.txt, so are being installed on read the docs. So maybe SciPy doesn't have to be mocked any more either.

josh146 · 2020-05-21T02:40:48Z

docs/conf.py

+def process_numba_signature(app, what, name, obj, options, signature, return_annotation):
+    if isinstance(obj, numba.targets.registry.CPUDispatcher):
+        original = obj.py_func
+        orig_sig = inspect.signature(original)
+
+        if (orig_sig.return_annotation) is inspect._empty:
+            ret_ann = None
+        else:
+            ret_ann = orig_sig.return_annotation.__name__
+
+        return (str(orig_sig), ret_ann)
+
+    return (signature, return_annotation)
+
+
 def setup(app):
+    app.connect('autodoc-process-signature', process_numba_signature)


Awesome that this works!

josh146 · 2020-05-21T04:08:48Z

https://github.com/numba/llvmlite/blob/c4f5a085c9f9c7a192579b02a11347bfc2ac768b/setup.py#L25

Looks like llvmlite disables installation on readthedocs environments :(

josh146 · 2020-05-21T04:19:55Z

@thisac: I updated the .readthedocs.yml file to use Python 3.6, and this avoids llvmlite/setup.py from running, since the binary wheel is available.

But RTDs then runs into a different error:

  File "/home/docs/checkouts/readthedocs.org/user_builds/hafnian/checkouts/fix_numba_decorator/docs/conf.py", line 345, in process_numba_signature
    if isinstance(obj, numba.targets.registry.CPUDispatcher):
AttributeError: module 'numba' has no attribute 'targets'

See the full build log here: https://readthedocs.org/projects/hafnian/builds/11086294/

Edit: could something have changed in the latest Numba release? What version of Numba do you have installed locally?

…nto fix_numba_decorator

thisac · 2020-05-21T18:19:41Z

@thisac: I updated the .readthedocs.yml file to use Python 3.6, and this avoids llvmlite/setup.py from running, since the binary wheel is available.

But RTDs then runs into a different error:
  File "/home/docs/checkouts/readthedocs.org/user_builds/hafnian/checkouts/fix_numba_decorator/docs/conf.py", line 345, in process_numba_signature
    if isinstance(obj, numba.targets.registry.CPUDispatcher):
AttributeError: module 'numba' has no attribute 'targets'
See the full build log here: https://readthedocs.org/projects/hafnian/builds/11086294/

Edit: could something have changed in the latest Numba release? What version of Numba do you have installed locally?

Thanks @josh146. I've fixed the AttributeError: module 'numba' has no attribute 'targets' by updating the requirements for Numba to numba>=0.49.1. I believe I could fix the mocking issue, although, as you mentioned, it might not be necessary, so I've left it as it is at the moment.

nquesada · 2020-05-21T21:20:45Z

hey @thisac : This is looking pretty good. There is one minor possibly related issue and is that the modules quantum and samples are no longer listed in The Walrus API:

josh146

Looks good @thisac, and nice fix! Note that you will need to add scipy to the list of documentation requirements so that it will build on read the docs (you can check the build console here: https://readthedocs.org/projects/hafnian/builds/).

Once the fix_numba_decorator is passing, feel free to merge this in 💯

josh146 · 2020-05-22T14:19:15Z

docs/conf.py

+    # 'numpy',
+    # 'numpy.dtype',
+    # 'scipy',
+    # 'scipy.special',
+    # 'scipy.optimize',
+    # 'scipy.stats',


Safe to delete all of these :)

docs/requirements.txt

Co-authored-by: Josh Izaac <josh146@gmail.com>

thisac added 2 commits May 20, 2020 21:27

Add numba signature fix

1ac41cd

Fix broken docstrings

c6bd379

thisac self-assigned this May 21, 2020

thisac requested review from josh146 and nquesada May 21, 2020 01:43

thisac added bug Something isn't working review-ready labels May 21, 2020

Update requirements.txt

2ee96d3

josh146 reviewed May 21, 2020

View reviewed changes

change readthedocs to use 3.6

4814a7d

thisac added 2 commits May 21, 2020 13:57

Update numba requirements

ac18725

Merge branch 'fix_numba_decorator' of github.com:XanaduAI/thewalrus i…

eccf9e6

…nto fix_numba_decorator

thisac added 2 commits May 22, 2020 10:01

Remove tabs

4e099af

Comment out scipy mocks

bb90d4c

thisac requested a review from josh146 May 22, 2020 14:06

josh146 approved these changes May 22, 2020

View reviewed changes

thisac and others added 2 commits May 22, 2020 10:25

Update docs/requirements.txt

a4343c5

Co-authored-by: Josh Izaac <josh146@gmail.com>

Remove mocked modules

8d203f7

thisac merged commit e1db968 into master May 22, 2020

thisac deleted the fix_numba_decorator branch May 22, 2020 15:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Numba decorated function docstrings #173

Fix Numba decorated function docstrings #173

thisac commented May 21, 2020 •

edited

Loading

codecov bot commented May 21, 2020 •

edited

Loading

josh146 left a comment

josh146 May 21, 2020

josh146 May 21, 2020

josh146 May 21, 2020

josh146 commented May 21, 2020

josh146 commented May 21, 2020 •

edited

Loading

thisac commented May 21, 2020

nquesada commented May 21, 2020

josh146 left a comment

josh146 May 22, 2020

thisac May 22, 2020

Fix Numba decorated function docstrings #173

Fix Numba decorated function docstrings #173

Conversation

thisac commented May 21, 2020 • edited Loading

codecov bot commented May 21, 2020 • edited Loading

Codecov Report

josh146 left a comment

Choose a reason for hiding this comment

josh146 May 21, 2020

Choose a reason for hiding this comment

josh146 May 21, 2020

Choose a reason for hiding this comment

josh146 May 21, 2020

Choose a reason for hiding this comment

josh146 commented May 21, 2020

josh146 commented May 21, 2020 • edited Loading

thisac commented May 21, 2020

nquesada commented May 21, 2020

josh146 left a comment

Choose a reason for hiding this comment

josh146 May 22, 2020

Choose a reason for hiding this comment

thisac May 22, 2020

Choose a reason for hiding this comment

thisac commented May 21, 2020 •

edited

Loading

codecov bot commented May 21, 2020 •

edited

Loading

josh146 commented May 21, 2020 •

edited

Loading