RFC: Source-based code coverage #3143
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
karkhaz
reviewed
Apr 16, 2024
celinval
reviewed
Apr 23, 2024
adpaco-aws
force-pushed
the
rfc-region-cov
branch
from
9c3e9ab
to
ae64a17
Compare
zhassan-aws
approved these changes
Aug 12, 2024
feliperodri
approved these changes
Aug 27, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Aug 27, 2024
This PR replaces the line-based coverage instrumentation we introduced in #2609 with the standard source-based code coverage instrumentation performed by the Rust compiler. As a result, we now insert code coverage checks in the `StatementKind::Coverage(..)` statements produced by the Rust compiler during compilation. These checks include coverage-relevant information[^note-internal] such as the coverage counter/expression they represent [^note-instrument]. Both the coverage metadata (`kanimap`) and coverage results (`kaniraw`) are saved into files after the verification stage. Unfortunately, we currently have a chicken-egg problem with this PR and #3121, where we introduce a tool named `kani-cov` to postprocess coverage results. As explained in #3143, `kani-cov` is expected to be an alias for the `cov` subcommand and provide most of the postprocessing features for coverage-related purposes. But, the tool will likely be introduced after this change. Therefore, we propose to temporarily print a list of the regions in each function with their associated coverage status (i.e., `COVERED` or `UNCOVERED`). ### Source-based code coverage: An example The main advantage of source-based coverage results is their precision with respect to the source code. The [Source-based Code Coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html) documentation explains more details about the LLVM coverage workflow and its different options. For example, let's take this Rust code: ```rust 1 fn _other_function() { 2 println!("Hello, world!"); 3 } 4 5 fn test_cov(val: u32) -> bool { 6 if val < 3 || val == 42 { 7 true 8 } else { 9 false 10 } 11 } 12 13 #[cfg_attr(kani, kani::proof)] 14 fn main() { 15 let test1 = test_cov(1); 16 let test2 = test_cov(2); 17 assert!(test1); 18 assert!(test2); 19 } ``` Compiling and running the program with `rustc` and the `-C instrument-coverage` flag, and using the LLVM tools can get us the following coverage result:  In contrast, the `cargo kani --coverage -Zsource-coverage` command currently generates: ``` src/main.rs (main) * 14:1 - 19:2 COVERED src/main.rs (test_cov) * 5:1 - 6:15 COVERED * 6:19 - 6:28 UNCOVERED * 7:9 - 7:13 COVERED * 9:9 - 9:14 UNCOVERED * 11:1 - 11:2 COVERED ``` which is a verification-based coverage result almost equivalent to the runtime coverage results. ### Benchmarking We have evaluated the performance impact of the instrumentation using the `kani-perf.sh` suite (14 benchmarks). For each test, we compare the average time to run standard verification against the average time to run verification with the source-based code coverage feature enabled[^note-line-evaluation]. The evaluation has been performed on an EC2 `m5a.4xlarge` instance running Ubuntu 22.04. The experimental data has been obtained by running the `kani-perf.sh` script 10 times for each version (`only verification` and `verification + coverage`), computing the average and standard deviation. We've split this data into `small` (tests taking 60s or less) and `large` (tests taking more than 60s) and drawn the two graphs below. #### Performance comparison - `small` benchmarks  #### Performance comparison - `large` benchmarks  #### Comments on performance Looking at the small tests, the performance impact seems negligible in such cases. The difference is more noticeable in the large tests, where the time to run verification and coverage can take 2x or even more. It wouldn't be surprising that, as programs become larger, the complexity of the coverage checking grows exponentially as well. However, since most verification jobs don't take longer than 30min (1800s), it's OK to say that coverage checking represents a 100-200% slowdown in the worst case w.r.t. standard verification. It's also worth noting a few other things: * The standard deviation remains similar in most cases, meaning that the coverage feature doesn't have an impact on their stability. * We haven't tried any SAT solvers other than the ones used by default for each benchmark. It's possible that other solvers perform better/worse with the coverage feature enabled. ### Call-outs * The soundness issue documented in #3441. * The issue with saving coverage mappings for non-reachable functions documented in #3445. * I've modified the test cases in `tests/coverage/` to test this feature. Since this technique is simpler, we don't need that many test cases. However, it's possible I've left some test cases which don't contribute much. Please let me know if you want to add/remove a test case. [^note-internal]: The coverage mappings can't be accessed through the StableMIR interface so we retrieve them through the internal API. [^note-instrument]: The instrumentation replaces certain counters with expressions based on other counters when possible to avoid a part of the runtime overhead. More details can be found [here](https://github.com/rust-lang/rustc-dev-guide/blob/master/src/llvm-coverage-instrumentation.md#mir-pass-instrumentcoverage). Unfortunately, we can't avoid instrumenting expressions at the moment. [^note-line-evaluation]: We have not compared performance against the line-based code coverage feature because it doesn't seem worth it. The line-based coverage feature is guaranteed to include more coverage checks than the source-based one for any function. In addition, source-based results are more precise than line-based ones. So this change represents both a quantitative and qualitative improvement. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 4, 2024
These are the auto-generated release notes: ## What's Changed * Update CBMC build instructions for Amazon Linux 2 by @tautschnig in #3431 * Handle intrinsics systematically by @artemagvanian in #3422 * Bump tests/perf/s2n-quic from `445f73b` to `ab9723a` by @dependabot in #3434 * Automatic cargo update to 2024-08-12 by @github-actions in #3433 * Actually apply CBMC patch by @tautschnig in #3436 * Update features/verify-rust-std branch by @feliperodri in #3435 * Add test related to issue 3432 by @celinval in #3439 * Implement memory initialization state copy functionality by @artemagvanian in #3350 * Bump tests/perf/s2n-quic from `ab9723a` to `80b93a7` by @dependabot in #3453 * Make points-to analysis handle all intrinsics explicitly by @artemagvanian in #3452 * Automatic cargo update to 2024-08-19 by @github-actions in #3450 * Add loop scanner to tool-scanner by @qinheping in #3443 * Avoid corner-cases by grouping instrumentation into basic blocks and using backward iteration by @artemagvanian in #3438 * Re-enabled hierarchical logs in the compiler by @celinval in #3449 * Fix ICE due to mishandling of Aggregate rvalue for raw pointers to `str` by @celinval in #3448 * Automatic cargo update to 2024-08-26 by @github-actions in #3459 * Bump tests/perf/s2n-quic from `80b93a7` to `8f7c04b` by @dependabot in #3460 * Update deny action by @zhassan-aws in #3461 * Basic support for memory initialization checks for unions by @artemagvanian in #3444 * Adjust test patterns so as not to check for trivial properties by @tautschnig in #3464 * Clarify comment in RFC Template by @carolynzech in #3462 * RFC: Source-based code coverage by @adpaco-aws in #3143 * Adopt Rust's source-based code coverage instrumentation by @adpaco-aws in #3119 * Upgrade toolchain to 08/28 by @jaisnan in #3454 * Extra tests and bug fixes to the delayed UB instrumentation by @artemagvanian in #3419 * Upgrade Toolchain to 8/29 by @carolynzech in #3468 * Automatic toolchain upgrade to nightly-2024-08-30 by @github-actions in #3469 * Extend name resolution to support qualified paths (Partial Fix) by @celinval in #3457 * Partially integrate uninit memory checks into `verify_std` by @artemagvanian in #3470 * Update Toolchain to 9/1 by @carolynzech in #3478 * Automatic cargo update to 2024-09-02 by @github-actions in #3480 * Bump tests/perf/s2n-quic from `8f7c04b` to `1ff3a9c` by @dependabot in #3481 * Automatic toolchain upgrade to nightly-2024-09-02 by @github-actions in #3479 * Automatic toolchain upgrade to nightly-2024-09-03 by @github-actions in #3482 * RFC for List Subcommand by @carolynzech in #3463 * Add tests for fixed issues. by @carolynzech in #3484 **Full Changelog**: kani-0.54.0...kani-0.55.0 By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upgrades the Kani coverage feature with the source-based code coverage implementation used in the Rust compiler.
Rendered version available here.
Related to #2640
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.