Separate build from execution & report generation

[MiniaczQ]

This is a mirror issue to another coverage tool xd009642/tarpaulin#1352

Is it possible to build binaries, then move them to special environments (like docker images) and run coverage and reports there separately?

[taiki-e]

I don't know your exact use case, but show-env + report might work: https://github.com/taiki-e/cargo-llvm-cov?tab=readme-ov-file#get-coverage-of-external-tests

[MiniaczQ]

The exact use case would be building all of the test binaries together, with cache available, then separating them to:

• unit tests, no special environment
• integration tests, docker image, docker compose with specific services available
  then combining all results through another tool

[taiki-e]

If you want to run the built binaries in containors you need to bind the target directory and map LLVM_PROFILE_FILE env var properly, but show-env + report should work.

[MiniaczQ]

Alright, I'll try it out and see how it goes.
Thank you :)

[taiki-e]

If you want to run the built binaries in containors you need to bind the target directory and map LLVM_PROFILE_FILE env var properly

I checked locally and something like the following worked:

source <(cargo llvm-cov show-env --export-prefix)

cargo build

docker run --rm -d -it \
  -v "$(pwd)":/workdir \
  --env LLVM_PROFILE_FILE="${LLVM_PROFILE_FILE/"$(pwd)"//workdir}" \
  ubuntu sh -c '/workdir/target/debug/<bin-name> --version'

cargo llvm-cov report --html

[MiniaczQ]

The assumption is that I provide all the test binaries paths, which requires build result parsing the build results.
I believe this is also different between cargo test and nextest

[taiki-e]

For cargo test you can get test binary paths while building the test binaries by calling cargo test --no-run --message-format=json and parsing the output JSON.

[MiniaczQ]

I did a setup with nextest, since it provides a better way to package test binaries, here's the attempt, it doesn't generate profraws and I'm not sure why
nextest-rs/nextest#1036

[taiki-e]

Closing -- this seems to be solved nextest-rs/nextest#1036 (comment)