-
-
Notifications
You must be signed in to change notification settings - Fork 180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tarpaulin fails to do coverage test on proc-macro repos #326
Comments
So cargo based issues can be caused by tarpaulin using an outdated version of cargo. The latest release 0.10.2 updated the cargo version and I can see your Travis run used 0.10.0. can you clear your cargo cache and try again to see if it works? |
I cleared cargo cache and rebuilt, the same issue persists |
Just tried it out on my machine and it worked fine, I also see there are some runs in your CI where it worked fine. Is there some set of feature flags etc I should be using to recreate the error?
|
Yes! From what I see, the problem is with only procedure macro repos, with a |
Ah so everything in the crate is a proc macro? I'll work on this my end but you might want to look at https://crates.io/crates/runtime-macros it turns proc macros evaluated at compile time into runtime expressions which means tarpaulin actually has something to cover. Though that would require tarpaulin getting past the Cargo.toml itself... |
Looks like the same issue with #336 |
have you tried clap_derive with the runtime-macros crate? That could be worth a shot |
Unfortunately, it doesn't test enough to be changed to. I could repeat the tests, but that's not DRY. |
So I moved my cargo dependency to the latest on master and it seems to solve the issue. You can try it out by installing from https://github.com/xd009642/tarpaulin/tree/issue_339 |
Could you do a pre-release so we can test it? I can't run it locally because I am on Mac. |
I'm at work now but I can do a docker build later and push that or you could always spin up a CI job to try it out in the meantime. I have a bigger change to the project that should solve this better by removing the cargo dependency so if you're more patient you can wait for that (I hope to have it done end of the weekend) |
I can wait. I had some issues with the docker build IIRC. Also, it was huge. Maybe you can look into https://github.com/docker-slim/docker-slim for the docker build. (I might even look into this today for you 😄) |
Still has issues.
|
This should be fixed in the 0.11.1 release, so I'm going to close this issue. If it's not sorted feel free to reopen and comment 👍 |
The trybuild tests are still failing. https://travis-ci.org/github/clap-rs/clap/jobs/662128329#L709 |
@pksunkara if you try the latest docker image for develop (or just do |
I don't see any changes in the develop branch that would fix this. But I still tried anyway at https://github.com/clap-rs/clap/pull/1747/files. It failed with the same thing, https://travis-ci.org/github/clap-rs/clap/jobs/662526678#L475. And running locally using docker is stuck on compiling step. (I waited for a long time).
Do you think you could forward the verbose flag to cargo? Maybe I will have a better idea if something is happening or not. |
That's because I'm on a different PC and it asked me for my git password to push and I didn't notice! Pushing the change now |
I'll add something so if you give |
Thanks. Still failing, https://travis-ci.org/github/clap-rs/clap/jobs/662526678#L544. I just restarted the job which installed the new develop HEAD |
So one of the issues is caused because the env var I think If you want the tests to run the same via |
You are talking about the non trybuild test failing? Yes, it's an issue with cargo env vars. Or are you saying this is a fix for the trybuild tests themselves? |
Yeah I'm talking about the trybuild test failures and this is how you could solve them without needing a change in tarpaulin. Personally for tests like that I'd prefer the compile time macros myself as it means the test runs the same each time |
Could you give more details on how to fix it? I am not sure I have the complete idea on how the fix works. |
I got the macro names wrong, found the right ones after a google. So in a test instead of let pkg_name = env::var("CARGO_PKG_NAME");
match pkg_name {
} Use https://doc.rust-lang.org/std/macro.option_env.html so it gets the var at compile time not run time let pkg_name = option_env!("CARGO_PKG_NAME");
match pkg_name {
} |
https://travis-ci.org/github/clap-rs/clap/jobs/662706650#L299 with clap-rs/clap@9b6eaae I even tried to do some intelligent stuff. Do you think this is an issue with |
@xd009642 Do you want a small reproducible repo with trybuild tests so you can work on it and fix it? (Either by fixing this crate or by sending PR to trybuild or by changing something in the project) |
Yeah sure I'll always accept reproducible examples 😊 |
I think this is due to |
Alternatively, tarpaulin could set |
@CreepySkeleton Thanks for pointing out. I started working on a preliminary probe to see if it actually fixes the issues and looks like it might. First attempt: Use https://travis-ci.org/github/clap-rs/clap/jobs/673138109 Some progress that now
|
I suspect this is because the name/dir you get though |
So, my earlier commit is completely wrong because we need those variables from runtime. |
@xd009642 Can you try setting these environment variables in |
@xd009642 Got it working here. I forked tarpaulin and added this commit which got it working. Unfortunately, I can't contribute a PR because I couldn't get tarpaulin working locally since I am on MacOS. But please, go ahead and take the commit and make a release fast. The only issue the commit might have is that I haven't ran it with |
Also, the coverage is 0 for the proc-macro crates as you can see here, https://coveralls.io/builds/29999944. The tests are passing now but coverage is not being correct. |
Yeah, we would need something like Can we just exclude the derive from coverage entirely? |
There's the |
I am okay with leaving them at 0 right now. But I am thinking of the long term future here where tarpaulin is the go to coverage tool for Rust. |
Now that the tests running is fixed, can someone give me a small bit of context on exactly why we can't get coverage of the proc-macros? |
To be fair, the issue is not exactly fixed, it's just ignored. Context (not a pro, feel free to correct me): this is what the process looks like:
The problem is that macro expansion only exists at the step one. Tarpaulin couldn't count it simply because the macros had already been expanded. They are not present in the binary, only the result of the expansion is, which cannot be mapped back to source. |
@CreepySkeleton that's spot on 👍 |
Tarpaulin would know if a package is proc-macro or not. Can we calculate coverage using interrupts for compilation (step one) itself? |
I doubt it. This would require squeezing the tarpaulin run between "the code have been parsed into a stream of tokens" and "macros have been expanded"... effectively someplace inside I think |
|
There's a PR awaiting to be landed 😉 But anyway, I don't understand your obsession with coverage. It is telling for |
I'm going to close this issue as the base issue is solved and the proc-macro stuff seems to be answered. |
I ran into the issue with testing proc-macro derives myself a while back and published |
Hi @xd009642, are there any other known issues similar to this? I added a proc-macro crate to a larger repository with many crates in it, and for some reason it is failing on It started failing coverage testing even before adding any tests to it. I added tests and it still failed, then I switched the test format to the one mentioned above, with I think its crashing the entire executable, not a particular test, but I can't tell, and I'm not sure how to get any additional information out of the logs through tarpaulin or flags to rust.
I understand you can't help much since I can't provide a repo as of yet, but I was just hoping you had seen something like this before. I skimmed the issues, but I could have missed it. Currently running:
|
I am not sure what to look or how to debug the issue but the error message is
It only appears in proc-macro repos.
Travis CI log for failing job with tarpaulin: https://travis-ci.org/Rair-Project/rair-core/jobs/638586153
Travis CI log for same job passing tests without tarpaulin https://travis-ci.org/Rair-Project/rair-core/builds/638586152
The text was updated successfully, but these errors were encountered: