Skip to content

Update documentation for coverage tests #1823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions src/llvm-coverage-instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,20 +273,22 @@ since it will not be called), and adds a new `FunctionCoverage`, with

## Testing LLVM Coverage

[(See also the compiletest documentation for the `tests/coverage-map` and
`tests/run-coverage` test suites.)](./tests/compiletest.md#coverage-tests)
[(See also the compiletest documentation for the `tests/coverage`
test suite.)](./tests/compiletest.md#coverage-tests)

Coverage instrumentation in the MIR is validated by a `mir-opt` test:
[`tests/mir-opt/instrument_coverage.rs`].

Coverage instrumentation in LLVM IR is validated by the [`tests/coverage-map`]
test suite. These tests compile a test program to LLVM IR assembly, and then
Coverage instrumentation in LLVM IR is validated by the [`tests/coverage`]
test suite in `coverage-map` mode.
These tests compile a test program to LLVM IR assembly, and then
use the [`src/tools/coverage-dump`] tool to extract and pretty-print the
coverage mappings that would be embedded in the final binary.

End-to-end testing of coverage instrumentation and coverage reporting is
performed by the [`tests/run-coverage`] and [`tests/run-coverage-rustdoc`]
test suites. These tests compile and run a test program with coverage
performed by the [`tests/coverage`] test suite in `coverage-run` mode,
and by the [`tests/coverage-run-rustdoc`] test suite.
These tests compile and run a test program with coverage
instrumentation, then use LLVM tools to convert the coverage data into a
human-readable coverage report.

Expand All @@ -296,19 +298,19 @@ expected LLVM IR instructions and structured data for a coverage-enabled
program, including various checks for Coverage Map-related metadata and the LLVM
intrinsic calls to increment the runtime counters.

Expected results for the `coverage-map`, `run-coverage`, `run-coverage-rustdoc`,
Expected results for the `coverage`, `coverage-run-rustdoc`,
and `mir-opt` tests can be refreshed by running:

```shell
./x test tests/*coverage* --bless
./x test coverage --bless
./x test coverage-run-rustdoc --bless
./x test tests/mir-opt --bless
```

[`tests/mir-opt/instrument_coverage.rs`]: https://github.com/rust-lang/rust/blob/master/tests/mir-opt/instrument_coverage.rs
[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map
[`tests/coverage`]: https://github.com/rust-lang/rust/tree/master/tests/coverage
[`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump
[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage
[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc
[`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc
[`coverage-llvmir`]: https://github.com/rust-lang/rust/tree/master/tests/run-make/coverage-llvmir

## Implementation Details of the `InstrumentCoverage` MIR Pass
Expand Down
47 changes: 33 additions & 14 deletions src/tests/compiletest.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,8 @@ The following test suites are available, with links for more information:
- `run-make-fulldeps` — `run-make` tests which require a linkable build of `rustc`,
or the rust demangler
- [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind
- [`coverage-map`](#coverage-tests) - tests for coverage maps produced by
coverage instrumentation
- [`run-coverage`](#coverage-tests) - tests that run an instrumented program
and check its coverage report
- [`run-coverage-rustdoc`](#coverage-tests) - coverage tests that also run
- [`coverage`](#coverage-tests) - tests for coverage instrumentation
- [`coverage-run-rustdoc`](#coverage-tests) - coverage tests that also run
instrumented doctests
- [Rustdoc tests](../rustdoc.md#tests):
- `rustdoc` — tests for rustdoc, making sure that the generated files
Expand Down Expand Up @@ -402,7 +399,28 @@ These may be removed in the future.

### Coverage tests

The tests in [`tests/coverage-map`] test the mappings between source code
The tests in [`tests/coverage`] are shared by multiple test modes that test
coverage instrumentation in different ways.
Running the `coverage` test suite will automatically run each test in all of
the different coverage modes.

Each mode also has an alias to run the coverage tests in just that mode:

```bash
./x test coverage # runs all of tests/coverage in all coverage modes
./x test tests/coverage # same as above

./x test tests/coverage/if.rs # runs the specified test in all coverage modes

./x test coverage-map # runs all of tests/coverage in "coverage-map" mode only
./x test coverage-run # runs all of tests/coverage in "coverage-run" mode only

./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only
```

---

In `coverage-map` mode, these tests verify the mappings between source code
regions and coverage counters that are emitted by LLVM.
They compile the test with `--emit=llvm-ir`,
then use a custom tool ([`src/tools/coverage-dump`])
Expand All @@ -416,18 +434,18 @@ coverage reports.

As a rule of thumb, any PR that doesn't change coverage-specific
code should **feel free to re-bless** the `coverage-map` tests as necessary,
without worrying about the actual changes, as long as the `run-coverage` tests
without worrying about the actual changes, as long as the `coverage-run` tests
still pass.

---

The tests in [`tests/run-coverage`] perform an end-to-end test of coverage reporting.
In `coverage-run` mode, these tests perform an end-to-end test of coverage reporting.
They compile a test program with coverage instrumentation, run that program to
produce raw coverage data, and then use LLVM tools to process that data into a
human-readable code coverage report.

Instrumented binaries need to be linked against the LLVM profiler runtime,
so `run-coverage` tests are **automatically skipped**
so `coverage-run` tests are **automatically skipped**
unless the profiler runtime is enabled in `config.toml`:

```toml
Expand All @@ -439,14 +457,15 @@ profiler = true
This also means that they typically don't run in PR CI jobs,
though they do run as part of the full set of CI jobs used for merging.

The tests in [`tests/run-coverage-rustdoc`] also run instrumented doctests and
---

The tests in [`tests/coverage-run-rustdoc`] also run instrumented doctests and
include them in the coverage report. This avoids having to build rustdoc when
only running the main `run-coverage` suite.
only running the main `coverage` suite.

[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map
[`tests/coverage`]: https://github.com/rust-lang/rust/tree/master/tests/coverage
[`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump
[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage
[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc
[`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc


## Building auxiliary crates
Expand Down