Report integration with Sphinx

[Paebbels]

Is your feature request related to a problem? Please describe.

Usually, all my Python projects are documented using Sphinx. Such a documentation contains also an auto generated Python Class Reference as well as multiple code quality indications / analysis reports. Currently this is: code coverage by coveragypy, unit tests by pytest, static typing analysis by mypy and documentation coverage by docstr_coverage.

Most tools provide a machine readable output format (e.g. in XML or JSON) and almost all are writing or converting them to an HTML report. This can be a single file or multiple files in a directory.

For integration into the overall documentation, multiple CI jobs generate the results and generated HTML pages and then a final CI job combines all artifacts to a combined static website for e.g. GitHub Pages.

In the last days I investigated and prototyped a new Sphinx extension, which integrates a documentation coverage report directly as native Sphinx content into the documentation. This offers some huge benefits:

• reports are part of the overall design / no style break
• reports are also part of other Sphinx backend outputs e.g. PDF documentation
• parts of the report can be referenced by the main documentation
• reports are part of the main navigation / table of content

I would like to discuss if and how to integrate coveragepy into such a Sphinx extension. My extension sphinx-reports is open to more report formats like code coverage.

This is an example from my prototype for documentation coverage:

Describe the solution you'd like
I would like to read and render code coverage results into Sphinx documentation pages.

I would like to address it in 2 steps:

1. Integrate coveragepy's summary page and then link to auto generated coverage pages.
   (I haven't figured out how to write complex Sphinx content like colored source code.)
2. Also write source code files via Sphinx.

The extensions alpha code is here: https://pytooling.github.io/sphinx-reports

Describe alternatives you've considered
I checked, if auto generated HTML code from coveragepy could be copied into a placeholder Sphinx page. While content was transferred, I found huge problems with CSS and Javascript contents. It would results in dozens - if not hundreds - of regex-based rewrite and transformation rules. This can't be maintained.

An other alternative would be coveragepy offering ReST (Restructured Text) output and such files could be referenced by a toctree directive.

Additional context
I believe that expressing the quality of an open-source project is a need to convince others to use and maybe invest time in someone open-source project. It also can shorten some discussions if a library can be used easily and safely.

[Paebbels]

Where can I find the documentation of status.json written into the HTML output directory. It looks like .... "nums": [0, 1, 246, 0, 105, 86, 3, 33], contains a summary of total, covered and uncovered elements.

[Paebbels]

Update: status.json doesn't contain the correct (or final) coverage values. I'm now switching to a full coverage file in JSON format.

Source: https://pytooling.github.io/sphinx-reports/coverage/index.html

[nedbat]

I think you've already worked this out, but status.json is not meant for external code to consume: it's a caching mechanism to speed up the HTML report. The coverage json command writes out data you can use.

I'm not sure the simplest way to get what you need. It seem to me like a niche need to have these reports included in Sphinx generated pages, so I want to find a path for you that doesn't change coverage too much.

[Paebbels]

As I transitioned to the "bigger" JSON, I'm already in the phase reading and using all coverage data. I hoped to get along with status.json as it's quite small compared to a full JSON or XML.

So I think ReST output or so from Coverage.py is not needed.

At next I want to figure out how to create subpages in Sphinx and how to colorize source code. This is currently black magic in Sphinx.

What might be interesting for me is the sorting feature that you use in your HTML tables.

Do you have suggestions on level for the coloring?
So what's bad, medium, good very good coverage?

[nedbat]

Hi, I don't have suggestions for the levels for various colors. Probably people will want it to be configurable, but that's up to you. It sounds like you have everything you need from me?

[Paebbels]

Probably people will want it to be configurable

Yes, the number of levels and the percentage can be configured. I plan to have some kind of default coloring, but otherwise the user colors are used. Actually it's a percentage, description text (good, bad, very bad, ..) and a CSS class name.

It sounds like you have everything you need from me?

May be this part:

What might be interesting for me is the sorting feature that you use in your HTML tables.

The Javascript and column sorting is this code written/owned by you or some kind of library? I think it might be good to have a hierarchy and a sorted view. I see Coverage.py add a JS file into the out put directory.