pytest-plugins / Sphinx extensions

[stefanv]

Hackmd link for the summit: https://hackmd.io/JL5slkxORA-q7VRN79v1sA

[tylerjereddy]

I haven't found a plugin that allows regex usage in i.e., pytest -k "test_name". That could be pretty useful, though not really exclusively useful to scientific projects.

This helps you avoid running:

• pytest -k "test_name_1"
• pytest -k "test_name_2"
• etc.

if the tests are spread over multiple modules but are affected by changes under iteration.

[pllim]

I am particularly interested to see if any of these can be moved upstream from Astropy. Or if Astropy could retire some of them and use something else that is better maintained and more mainstream.

pytest

• https://github.com/astropy/pytest-arraydiff
• https://github.com/astropy/pytest-doctestplus
• https://github.com/astropy/pytest-filter-subpackage
• https://github.com/astropy/pytest-remotedata

Sphinx

• https://github.com/astropy/astropy-sphinx-theme
• https://github.com/astropy/sphinx-automodapi

[bsipocz]

Tyler's feature request rings a bell (not necessarily regex, but to accept wildcard), but now I can't seem to find it. Point is that it was certainly asked as a feature before and I'm sure would be appreciated by many libraries.

[bsipocz]

And I for one will have more incentive and thus time to maintain the plugins above (with the exception of the sphinx-theme, that is extremely old-fashioned and should have been retired a long ago) if: - they are move out from the astropy organization somewhere where infrastructure and its maintenance is valued. - have a wider adoption than a single domain.

[pllim]

Re: #10 (comment) -- I agree that for wider adoption, they should be moved out of Astropy org since these are not strictly related to astronomy sub-field in science. I will bring this issue to Astropy infrastructure tag-up but I do not foresee any opposition. Thanks!

[pllim]

Re: astropy theme -- I think Astropy would be really happy to maintain once less theme if we can get a new one to work. 😸 Please also see astropy/astropy-sphinx-theme#29

[drammock]

commenting because I can't self-assign. MNE-Python has a couple small sphinx extensions that might be of general interest:

• this one turns :gh:\`1234\` into a link to the issue/PR
• this one turns :newcontrib:\`Name Surname\` into a bold-formatted link and prepends "new contributor" to it
• this one joins a number and its unit with a non-breaking space so they aren't separated across linebreaks.

[bsipocz]

Neither plugin nor extension, but numpydoc is also very much part of the shared infrastructure.

And fixing the single-vs-double backtick issues for parameters in numpydoc would be a nice outcome that helps with doctesting, too: numpy/numpydoc#419 (comment)

[rossbar]

And fixing the single-vs-double backtick issues for parameters in numpydoc

For context, this often rears it's ugly head when sphinx's nitpicky mode is activated, as the parameters enclosed in single backticks (as recommended by the numpydoc style guide) then get wrapped with the default sphinx role (typically :py:obj:), creating broken links. This is definitely on my list of todo's for the summit!

[tylerjereddy]

FYI, I drafted a pytest-regex plugin in 2-3 hours or so, which already seems considerably more capable than the conventional -k. Of course there are no tests for the plugin itself yet, but I added examples on how to use it in the README over there, and it seems to do the right thing so far...

Anyway, just getting rid of the initial inertia on that ahead of the summit. FWIW, it sounds like the main pytest project may not be opposed to directly supporting it from one or two issues I scanned through, but prototying quickly externally at the summit that way may not hurt anyway, and perhaps they'd handle the absorption if they really wanted.

The other thing is that folks may want features that the main project wouldn't, so defining that first probably makes sense with a test in a very small external plugin codebase to start.

[seberg]

One thing related that I would be interested in also is infrastructure similar to/tagged on top of pytest-doctestplus to modify docs based on actual output. (That is to enable mass changes, I suppose whatever that should be should have something like a new_str = func(old_str, evaluated_str) to hook that in.)

(That is related to the fact we need to change how NumPy scalars print.)

[bsipocz]

@seberg - There is nothing new under the sun, would be wonderful to implement this issue 😅 scientific-python/pytest-doctestplus#107

[drammock]

Another "neither pytest nor sphinx" thing that could be standardized/shared across projects is the website banner saying "warning this is an old version / the development version of this package, please switch to stable docs". The pydata-sphinx-theme has added an announcement banner that is configured in conf.py but it's not easy to get it to contextually appear/change messages depending on docs version (esp. when older versions of your docs were built with other themes / before the announcement banner was part of the theme).

• @jarrodmillman describes the networkx approach here
• MNE-Python uses this javascript that was originally adapted from sklearn but has diverged somwhat over the years

What are other folks doing? Is it worth trying to standardize/centralize this for the ecosystem?

[stefanv]

Yes, please! I think this should be a built-in feature in pydata-sphinx-theme. It's not too hard to implement, but there are a few subtleties as you'll see on that issue on skimage

[tylerjereddy]

I touched base with upstream about pytest-regex in the cross-linked issue above, and so far it sounds like they won't want to upstream it--they may pursue some kind of JSON parsing technology for test selection. It doesn't seem like it is worth arguing with them about that, it was so easy to implement maybe it should just be a tiny plugin then, not sure how much time it is worth spending on though, maybe just try to get a few tests and put on PyPI and see if anyone uses it.