Generate coverage reports externally, no cargo binary

[testQx]

One scenario I have is a program written in rust that exposes the http interface and does not generate Python-related dependency packages via pyo3.
Invoke http interface directly by executing request through pytest;

Is it possible for me to get a coverage report for this integration test?

Trying the following method so far does not generate the relevant *.profraw

In simple terms, for example, rust program is a back-end, it has a front-end page, can I get the coverage of the relevant operations through the front-end operations

# Set the environment variables needed to get coverage.
source <(cargo llvm-cov show-env --export-prefix)
# Remove artifacts that may affect the coverage results.
# This command should be called after show-env.
cargo llvm-cov clean --workspace
# Above two commands should be called before build binaries.

cargo build # Build rust binaries.
# Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommands.
#...

cargo llvm-cov report --lcov # Generate report without tests.

[taiki-e]

Trying the following method so far does not generate the relevant *.profraw

cargo build # Build rust binaries.
# Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommand.
#...

After you built the binaries, did you run them? If you have not run any, no profiles will be generated. If you run the binary, how exactly did you do it? (When using tools that do not propagate environment variables or parent directory structures, such as docker for example, environment variables must be propagated/mapped properly. See also #319 (comment))

[testQx]

Trying the following method so far does not generate the relevant *.profraw

cargo build # Build rust binaries.
# Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommand.
#...

After you built the binaries, did you run them? If you have not run any, no profiles will be generated. If you run the binary, how exactly did you do it? (When using tools that do not propagate environment variables or parent directory structures, such as docker for example, environment variables must be propagated/mapped properly. See also #319 (comment))

My steps are as follows:

1. source through the environment variable generated in cargo llvm-cov show-env --export-prefix
2. The cargo llvm-cov clean --workspace is executed
3. cargo build --release --features tls Compile my side of the rust project
4. Run./target/release/xxx
5. Run pytest to execute the http request call
6. kill -9 Indicates related service processes
7. Run the cargo llvm-cov report --lcov

But there is no *.profraw file

[taiki-e]

• kill -9 Indicates related service processes

AFAIK, SIGKILL is currently not supported. See #235 for more.

[testQx]

Trying the following method so far does not generate the relevant *.profraw

cargo build # Build rust binaries.
# Commands using binaries in target/debug/*, including `cargo test` and other cargo subcommand.
#...

After you built the binaries, did you run them? If you have not run any, no profiles will be generated. If you run the binary, how exactly did you do it? (When using tools that do not propagate environment variables or parent directory structures, such as docker for example, environment variables must be propagated/mapped properly. See also #319 (comment))

I found a problem,
rust services run on physical machines
But I mount the working directory through docker and do the steps 1,2,3 above, and there is no cargo command on the physical machine.
Then I perform Step 4 on the physical machine.
Do environment variables need to be set on the physical machine to run the program? Whether the environment variable set here is the same as the docker internal, using a relative path, or the external physical machine absolute path address

[testQx]

• kill -9 Indicates related service processes

AFAIK, SIGKILL is currently not supported. See #235 for more.

*.profraw file

Is the *.profraw file generated when I run pytest (while the rust program is still working), and how do I trigger the *.profraw file

[taiki-e]

Do environment variables need to be set on the physical machine to run the program? Whether the environment variable set here is the same as the docker internal, using a relative path, or the external physical machine absolute path address

set-env currently sets the following environment variables:

export RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target'
export LLVM_PROFILE_FILE='/path/to/target/name-%p-%10m.profraw'
export CARGO_LLVM_COV=1
export CARGO_LLVM_COV_SHOW_ENV=1
export CARGO_LLVM_COV_TARGET_DIR=/path/to/target

The environment variables actually used in each step are different.

• cargo build requires RUSTFLAGS.
• Binaries require LLVM_PROFILE_FILE.
• cargo llvm-cov report and cargo llvm-cov clean require CARGO_LLVM_COV_TARGET_DIR.

When using docker or other, absolute paths may need to be mapped to point to the same directory. (LLVM_PROFILE_FILE or/and CARGO_LLVM_COV_TARGET_DIR, depending on the situation)

Is the *.profraw file generated when I run pytest (while the rust program is still working), and how do I trigger the *.profraw file

IIRC, rust binary (or cdylib) will generate *.profraw file on exit.
The LLVM documentation says there is a way to generate file on a regular basis, but it didn't work when I tried it before. (#280)

[testQx]

Do environment variables need to be set on the physical machine to run the program? Whether the environment variable set here is the same as the docker internal, using a relative path, or the external physical machine absolute path address

set-env currently sets the following environment variables:

export RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target'
export LLVM_PROFILE_FILE='/path/to/target/name-%p-%10m.profraw'
export CARGO_LLVM_COV=1
export CARGO_LLVM_COV_SHOW_ENV=1
export CARGO_LLVM_COV_TARGET_DIR=/path/to/target

The environment variables actually used in each step are different.

• cargo build requires RUSTFLAGS.
• Binaries require LLVM_PROFILE_FILE.
• cargo llvm-cov report and cargo llvm-cov clean require CARGO_LLVM_COV_TARGET_DIR.

When using docker or other, absolute paths may need to be mapped to point to the same directory. (LLVM_PROFILE_FILE or/and CARGO_LLVM_COV_TARGET_DIR, depending on the situation)

Is the *.profraw file generated when I run pytest (while the rust program is still working), and how do I trigger the *.profraw file

IIRC, rust binary (or cdylib) will generate *.profraw file on exit. The LLVM documentation says there is a way to generate file on a regular basis, but it didn't work when I tried it before. (#280)

I tried using LLVM_PROFILE_FILE, but it didn't work.
If I inject so file at compile time, this so file is mainly used to listen to semaphore, can I generate *.profraw file, which method can I trigger internally to generate?

[Ekleog-NEAR]

I have hit a similar issue, in this CI run.

As can be seen there, cargo llvm-cov show-env sets LLVM_PROFILE_FILE to '/home/runner/work/nearcore/nearcore/target/nearcore-%p-%16m.profraw'. Yet cargo llvm-cov report is searching in another folder:

warning: not found *.profraw files in /home/runner/work/nearcore/nearcore/target/llvm-cov-target

I thought this might be a bug in show-env, and I tried solving it by adding this line to CI, after the show-env:

      - run: echo "LLVM_PROFILE_FILE=$(pwd)/target/llvm-cov-target/nearcore-%p-%16m.profraw" >> "$GITHUB_ENV"

However, this resulted in this other failing CI job, where the profraw was still not detected.

I then tried to add back the clean step, that I was trying to avoid in order to avoid spending tens of minutes of CI time rebuilding dependencies all the time. It still failed, and find found no *.profraw files.

The workflow I’m currently trying to get to work is the following:
https://github.com/near/nearcore/pull/10180/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR79-R102

Do you have any idea what I’m doing wrong? AFAICT with the last iteration, I’m basically only trying to get the cargo llvm-cov show-env-based example to work, without anything too specific to our setup.

[taiki-e]

@Ekleog-NEAR One of the problems is that you are passing the output of --export-prefix to GITHUB_ENV; -export-prefix adds a prefix for the UNIX shell, which is not the format GITHUB_ENV expects.

One of the ways is to save the output of --export-prefix to a file and source it, as in nextest-rs/nextest#1036.

[Ekleog-NEAR]

Ooooh now I feel stupid for copy-pasting without checking the documentation ._. Thank you for the solution! I ended up with this code for setting the environment in a github action. Would you be interested in a PR that showcases the more "advanced" (multiple codecov labels, using with python tests that call into rust, etc.) use cases of cargo-llvm-cov in the readme?

Also I’m curious, does llvm-cov actually need the whole target folder, or would it be possible to trim it somehow, to be able to share build artifacts across jobs without having to share the whole target folder? And if it’s possible (and significantly reduces the size of the target folder), would you be interested in a subcommand that cleans target to just leave the necessary stuff?

Also, just to say, I had parsed --export-prefix as "also export the prefix, like the target/ directory", resulting in this mistake. Maybe on the next major upgrade of cargo-llvm-cov, you could consider renaming it into --with-export-prefix, that does not have this potential for mis-parsing? (I’d totally understand if the churn would be too big, it’s just an idea)

[taiki-e]

Would you be interested in a PR that showcases the more "advanced" (multiple codecov labels, using with python tests that call into rust, etc.) use cases of cargo-llvm-cov in the readme?

A PR to add an example is welcome!

Also I’m curious, does llvm-cov actually need the whole target folder, or would it be possible to trim it somehow, to be able to share build artifacts across jobs without having to share the whole target folder? And if it’s possible (and significantly reduces the size of the target folder), would you be interested in a subcommand that cleans target to just leave the necessary stuff?

Only the binaries and profraw are actually needed to generate the report.
I think it would be okay to add a subcommand that copies some of the files that clean_partial deletes instead of deleting and outputs to a specified directory and an option to get the needed files from the directory containing the output of that command instead of the target directory.

Also, just to say, I had parsed --export-prefix as "also export the prefix, like the target/ directory", resulting in this mistake. Maybe on the next major upgrade of cargo-llvm-cov, you could consider renaming it into --with-export-prefix, that does not have this potential for mis-parsing? (I’d totally understand if the churn would be too big, it’s just an idea)

I have no strong opinion on the naming. That said, it may indeed be clearer to clarify that "export" is a noun, not a verb.