Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect Coverage Statistics when using a Workspace #1281

Open
Javagedes opened this issue May 10, 2023 · 4 comments
Open

Incorrect Coverage Statistics when using a Workspace #1281

Javagedes opened this issue May 10, 2023 · 4 comments
Assignees

Comments

@Javagedes
Copy link

Javagedes commented May 10, 2023

Describe the bug
Hello, unless I missed a command flag, tarpaulin does not take into account which package is being tested using the -p flag when it comes to calculating coverage changed. What I mean by this is running cargo tarpaulin twice, each with a different package, will result in an incorrect calculation for change in coverage because it is comparing the coverage difference between the first package and the second package

As an example:

`cargo tarpaulin -p Package1

To Reproduce

cargo tarpaulin -p pkg1
16.08% coverage, 205/1275 lines covered, +0.00% change in coverage

cargo tarpaulin -p pkg1
16.08% coverage, 205/1275 lines covered, +0.00% change in coverage

cargo tarpaulin -p pkg2
13.08% coverage, 165/1261 lines covered, -2.99% change in coverage

Expected behavior
I would expect information to be cached based on the package, and comparisons be made on a per package basis (or workspace as a whole without -p) when determining coverage changes.

@xd009642
Copy link
Owner

So I think currently my implementation is rather naive in it stores the previous coverage results but no details of the configuration and then just compares the cached previous run and current run. I can add something to store config info in that dat aand check it and not show the delta if the runs don't match.

I probably won't get onto it until next week but if you (or someone else) is interested in contributing most of the work would be in src/report/mod.rs in the print_summary function. Probably want to change the serialised output from TraceMap to some struct containing TraceMap and Config, update the saving and loading to add a check.

@Javagedes
Copy link
Author

Javagedes commented May 11, 2023

@xd009642 Thank you! I think I should have some time in the next week or two to work on it, and it's not a big rush for us, so if you have more urgent matters, you can wait for me to put up a PR for this and review it. Thanks for pointing me in the right direction, it will make getting started easier!

@wookietreiber
Copy link

Also note that the report/results include files from the other package:

# Cargo.toml
[workspace]
members = ["lib", "cli"]
$ cargo tarpaulin -p lib
...
Jul 15 14:52:21.192  INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| lib/src/foo.rs: 56/56 +0.00%
|| lib/src/bar.rs: 27/27 +0.00%
|| cli/src/config.rs: 0/2 +0.00%
|| cli/src/log.rs: 0/6 +0.00%
|| cli/src/main.rs: 0/59 +0.00%
||
55.33% coverage, 83/150 lines covered, +0.00% change in coverage

For the moment, I only really care about the coverage in the lib workspace member. I did not expect the files in cli to show up with -p lib, and I would not like them to count.

@wookietreiber
Copy link

wookietreiber commented Jul 15, 2023

For the moment, I only really care about the coverage in the lib workspace member. I did not expect the files in cli to show up with -p lib, and I would not like them to count.

Going one step further, it might also make sense to only consider the lines in lib for the coverage report and use the code in cli to detect coverage, i.e. as if cli was a yet another bin that'd count with --run-types. It would certainly be nice to be able to specify this on the command line, e.g.:

cargo tarpaulin --package lib --run-types ...,Workspace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants