Exporting coverage data to Cobertura XML

[nickg]

@qarlosalberto asked on LinkedIn about integrating the coverage output from NVC into TerosHDL. At the moment it's not possible to export coverage data from NVC's proprietary format into other tools so I added a command to generate Cobertura XML. This is quite a popular format which allows import into GitLab and other CI systems that support code coverage. I've tested with pycobertura but it should work with any other tool. However the format doesn't support hardware-specific features like toggle coverage or separating coverage by design hierarchy so we probably want to add UCIS XML at some point too. That would allow us to directly plug-in to pyucis, pyEDAA.UCIS, etc and even import into RiveraPro and other tools that support the UCIS interchange format.

You can export the data with nvc -c --export=out.xml path/to/file.covdb but I feel the nvc -c command is a bit difficult and non-intuitive to use at the moment so I experimented with also adding a standalone command nvc --cover-export --format=cobertura -o out.xml toplevel where toplevel is the top-level unit name as with -e, -r, etc. This way you don't have to know the path to the .covdb file (which is sort-of private anyway as it's within the library directory) and the options are easier to understand as the --cover-export options are grouped separately rather than being mixed in with the other -c options. It's also possible to chain this command with others e.g. nvc -e --cover toplevel -r --cover-export --format=cobertura -o out.xml. I'm considering replacing -c with --cover-report which would also generate the .covdb file name automatically from the top-level name. E.g. nvc --cover-report toplevel --html=html/ --exclude=.... @Blebowski?

[Blebowski]

Hi @nickg,

Wow, Cobertura export looks cool :) I agree that UCIS XML is usefull.

As for the top_level instead of .covdb file explicitly, I am not sure. I agree that the .covdb is sort-of private thing since it is placed to the work folder where rest of the code is compiled. So, user does not really need to see the .covdb. OTOH, user always needs to see different "runs" of the simulator to allow merging coverage from different runs. AFAIK, coverage report from a single test run is almost useless. Most of commercial grade TBs (based on any methodology UVM, Vunit...) require different invocations of the simulator to run different tests. Then, user (or framework using the simulator) needs to somehow pass location of the collected coverage data from individual runs. Whether user passes directly the .covdb file, or the folder where all the code is compiled/elaborated, it does not matter.

The only use-case where I can think of exporting UCIS/XML from single test run, is if there was a way how to merge the coverage from various UCIS/XML files in some external tool.

I am not sure how to do this with toplevel syntax only. When TB is the same, only test-cases change, then name of toplevel will be the same.

I am not sure about splitting the --cover-export from -c. It seems to me that keeping single command for "dealing with coverage" is simpler, but that is just one person opinion. -c is little non-intuitive though.

WBU following format:

nvc (-c|--coverage) [--merge <folder>] [--html-report-dir <folder>] [--export <file> --format=(cobertura|ucis)]  [<folder_with_run_1> [<folder_with_run_2> ... ]]

Similar to this topic, (but reverse), is tracking in which test a coverage item was covered. When someone merges coverage from e.g. 20 different tests, it would be handy somehow to remember the source covdb/run-folder, so that user can get something like:
"Yay, this nasty corner-case is really only covered in a dedicated tests I wrote for that :)"

[nickg]

I made the change to split the report/merge commands in f1dfe97. IMO it's much simpler and easier to use as two commands. E.g.

nvc --cover-merge -o out.covdb in1.covdb in2.covdb in3.covdb
nvc --cover-report -o html in1.covdb in2.covdb

[Blebowski]

Thanks. It seemed to me that the --cover-export and --cover was not really consistent. I was thinking to merge it into -c as --export, but your approach, have one command per task, is better.

[nickg]

I am not sure how to do this with toplevel syntax only. When TB is the same, only test-cases change, then name of toplevel will be the same.

On #507 I suggested adding a --name elaboration option which would let the user rename the elaborated design, I think that would also be useful here. The use cases seem to be:

1. Run the same top-level multiple times without re-elaborating (e.g. test case is read from a file). The counts just accumulate in the .covdb file so this works as-is.
2. Run the same top-level multiple times but re-elaborate each time (e.g. with different generics). This needs the --name option to avoid clobbering the previous results.
3. Different top-levels but some shared hierarchy.

For 2 and 3 there needs to be a way to merge results, but that can be done by listing multiple top-levels e.g. --cover-report top1 top2 top2.

[Blebowski]

I agree that --name is probably the best way to solve this. For the case 1, I don't think it works like that currently, IIRC, the covdb is overwritten. But that is an easy thing to change.