Add workflow code coverage report and upload

[jtmoon79]

Add github workflow to run grcov code coverage over cargo test and upload report to codecov.io

Here is the report for this commit https://app.codecov.io/github/chronotope/chrono/commit/b46ac6c090f1832e1c73903318098067d471737b/tree
Comparison data is missing because there have been no prior code coverage runs. This URL can be found in workflow grcov, step upload codecov, in the last few logged lines. e.g.

[2023-07-09T06:36:32.407Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/chronotope/chrono/commit/0e251e6b2764ec2f736300f925b5e22b09c58bd4"}

The README.md also has a badge+link to the latest codecov.io code coverage report.

For some reason, the newly added grcov workflow does not show up in some Actions views. I can find it by clicking Actions (at the top), then click grcov (left-side), then the runs of grcov.yml are listed.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (0.4.x@56c6967). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 6c4c1af differs from pull request most recent head 6db2eb9. Consider uploading reports for the commit 6db2eb9 to get more accurate results

@@           Coverage Diff            @@
##             0.4.x    #1178   +/-   ##
========================================
  Coverage         ?   88.90%           
========================================
  Files            ?       30           
  Lines            ?    11489           
  Branches         ?        0           
========================================
  Hits             ?    10214           
  Misses           ?     1275           
  Partials         ?        0           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[pitdicker]

How does this compare to cargo-llvm-cov? https://github.com/taiki-e/cargo-llvm-cov#continuous-integration

[jtmoon79]

I added cargo-llvm-cov job to my own project to "test drive" it.

cargo-llvm-cov and coveralls.io

Here's my code for a cargo-llvm-cov job https://github.com/jtmoon79/super-speedy-syslog-searcher/blob/9948f0a9035b0883644f0a37d63d16a77158be5b/.github/workflows/rust.yml#L543-L583

Run time is 8m10s. https://github.com/jtmoon79/super-speedy-syslog-searcher/actions/runs/5516276662

Coverage reported is 58%

I tried out coveralls.io as a renderer service. Here is an example of the output https://coveralls.io/builds/61257604

grcov and codecov.io

Here's my code for grcov job https://github.com/jtmoon79/super-speedy-syslog-searcher/blob/9948f0a9035b0883644f0a37d63d16a77158be5b/.github/workflows/rust.yml#L445-L540 (similar to this PR)

Run time is 23m59s. https://github.com/jtmoon79/super-speedy-syslog-searcher/actions/runs/5516276662

Coverage reported is 54%

Here is the codecov.io rendering of that output https://app.codecov.io/gh/jtmoon79/super-speedy-syslog-searcher/commit/93a58cc0254e8f4965fe7e3d5cd702489a237ee0/tree

I don't currently know why the coverage differences are different between cargo-llvm-cov and grcov.

Which one?

Which do you prefer @pitdicker @djc ?

cargo-llvm-cov is simpler to setup and run, and is faster than grcov.

I found the rendering at coveralls.io better looking and a bit easier to navigate. Both are decent, and AFAIK, permanently free for open-source projects.

[pitdicker]

Great you looked at both! The highlighted lines in coveralls.io make slightly more sense to me.

I suppose it is the same as with benchmarking: you can spend a lot of time trying to get more accurate. But really the useful part is doing a measurement and having something to compare against.

The CI run for with grcov for this PR took ca. 5 minutes (https://github.com/chronotope/chrono/actions/runs/5498456361/jobs/10019932092?pr=1178), most of which is spend in cargo install grcov. It seems the runtime does not have to be a concern for us? Although it is nice cargo-llvm-cov is the faster one.

[pitdicker]

Is one of them easier to replicate 'at home'? And do we have a place to document how to do so?

[jtmoon79]

I suppose it is the same as with benchmarking: you can spend a lot of time trying to get more accurate. But really the useful part is doing a measurement and having something to compare against.

Yep.

Is one of them easier to replicate 'at home'? And do we have a place to document how to do so?

I think you're asking "which is easier to use?" ? If so, then cargo-llvm-cov is definitely easier, as you can see from the amount of setup in each github Action job.

[djc]

I prefer cargo-llvm-cov over grcov. It is my impression that coveralls is no longer being actively developed? CodeCov hasn't exactly been without issues but it was recently acquired by Sentry who I have a decent amount of trust in.

[pitdicker]

I have no idea about either, but https://coveralls.io/announcements seems pretty active.

[jtmoon79]

I changed to use llvm-cov and codecov.io

See

• https://app.codecov.io/github/chronotope/chrono/commits?branch=All%20branches
• https://app.codecov.io/github/chronotope/chrono/tree/jtmoon79_add_code_coverage

[jtmoon79]

https://app.codecov.io/github/chronotope/chrono/commit/6db2eb94a29243286b783c614ac22fd216e058ba

[djc]

Thanks!