Skip to content

Commit 548d89f

Browse files
committed
maint/CI ~ convert to instrumented/source-based coverage calculation
## [why] Coverage calculations using the past `grcov` method has become increasingly ... odd, with strange lapses not obviously related to code changes. More recently, Rust (and `grcov`) have been promoting a new method for code coverage calculation, called instrumented (or source-based). Discussion about instrumented coverage has now been stabilized and included in [_The rustc book_][^1]. Unfortunately, the current incarnation of instrumented coverage does not support branch coverage calculation. It's, at best, a work-in-progress, possibly a works-as-designed, with, currently, a lack of strong support for adding branch support. "Region" coverage calculation is instead being discussed as a substitute for branch calculations. Ultimately, given increasing flakiness with the current method, a more robust and accurate method of line coverage is worth the loss of information arising from lack of branch coverage, which is of lower importance for most development. ### refs [^1]: https://doc.rust-lang.org/rustc/instrument-coverage.html - [Rust ~ Instrumentation-based coverage](https://doc.rust-lang.org/rustc/instrument-coverage.html) @@ <https://archive.is/ERWrk> - [HowTo collect Rust source-based coverage](https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html) @@ <https://archive.is/X9R14> - [`grcov` issue ~ missing branch coverage](mozilla/grcov#520) - [Rust issue ~ add support for branch coverage](rust-lang/rust#79649)
1 parent d2dc0d1 commit 548d89f

File tree

1 file changed

+37
-12
lines changed

1 file changed

+37
-12
lines changed

.github/workflows/ci.yml

+37-12
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
# spell-checker:ignore (rust) clippy rustfmt rustup RUSTC RUSTFLAGS Zpanic Cpanic RUSTDOCFLAGS Ccodegen Coverflow
2-
# spell-checker:ignore (bash) alnum esac (jargon) maint (utils) codecov grcov lcov (vars) tempfile () ntempfile
1+
# spell-checker:ignore (rust) clippy rustfmt rustup RUSTC RUSTFLAGS Zpanic Cpanic RUSTDOCFLAGS Ccodegen Coverflow profraw
2+
# spell-checker:ignore (abbrevs/acronyms) MSVC (bash) alnum esac (jargon) maint (utils) codecov grcov lcov (vars) tempfile () ntempfile
33

44
on: [push, pull_request]
55

@@ -97,16 +97,16 @@ jobs:
9797
- { os: windows-latest, features: windows }
9898
steps:
9999
- uses: actions/checkout@v3
100-
- name: Initialize workflow variables
100+
- name: Initialize/setup workflow
101101
id: vars
102102
shell: bash
103103
run: |
104-
## VARs setup
104+
## VARs and paths setup
105105
outputs() { step_id="${{ github.action }}"; for var in "$@" ; do echo steps.${step_id}.outputs.${var}="${!var}"; echo "${var}=${!var}" >> $GITHUB_OUTPUT; done; }
106106
# toolchain
107107
TOOLCHAIN="nightly" ## default to "nightly" toolchain (required for certain required unstable compiler flags) ## !maint: refactor when stable channel has needed support
108108
# * specify gnu-type TOOLCHAIN for windows; `grcov` requires gnu-style code coverage data files
109-
case ${{ matrix.job.os }} in windows-*) TOOLCHAIN="$TOOLCHAIN-x86_64-pc-windows-gnu" ;; esac;
109+
case ${{ matrix.job.os }} in windows-*) TOOLCHAIN="$TOOLCHAIN-x86_64-pc-windows-msvc" ;; esac;
110110
# * use requested TOOLCHAIN if specified
111111
if [ -n "${{ matrix.job.toolchain }}" ]; then TOOLCHAIN="${{ matrix.job.toolchain }}" ; fi
112112
outputs TOOLCHAIN
@@ -116,6 +116,13 @@ jobs:
116116
# * CODECOV_FLAGS
117117
CODECOV_FLAGS=$( echo "${{ matrix.job.os }}" | sed 's/[^[:alnum:]]/_/g' )
118118
outputs CODECOV_FLAGS
119+
# * code coverage
120+
COVERAGE_REPORT_DIR="target/debug/coverage"
121+
PROFILES_DIR="${COVERAGE_REPORT_DIR}/profiles"
122+
mkdir -p "${COVERAGE_REPORT_DIR}" "${PROFILES_DIR}"
123+
outputs COVERAGE_REPORT_DIR PROFILES_DIR
124+
LLVM_PROFILE_FILE="${PROFILES_DIR}/profile-%p-%m.profraw"
125+
outputs LLVM_PROFILE_FILE
119126
- name: Install `rust` toolchain
120127
shell: bash
121128
env:
@@ -124,31 +131,49 @@ jobs:
124131
## Install `rust` toolchain
125132
rm -f "${HOME}/.cargo/bin/"{rustfmt,cargo-fmt}
126133
rustup toolchain install ${TOOLCHAIN} -c rustfmt --profile minimal
134+
rustup component add --toolchain ${TOOLCHAIN} llvm-tools
127135
rustup default ${TOOLCHAIN}
128136
- run: cargo install grcov
129137
- name: Test
130138
shell: bash
131139
env:
132140
CARGO_INCREMENTAL: '0'
141+
LLVM_PROFILE_FILE: ${{ steps.vars.outputs.LLVM_PROFILE_FILE }}
133142
RUSTC_WRAPPER: ''
134-
RUSTFLAGS: '-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort'
135-
RUSTDOCFLAGS: '-Cpanic=abort'
143+
RUSTFLAGS: '-C instrument-coverage'
144+
RUSTDOCFLAGS: ''
136145
run: |
137146
## Test
138147
cargo test ${{ steps.vars.outputs.CARGO_FEATURES_OPTION }} --no-fail-fast
148+
- name: Test examples
149+
shell: bash
150+
env:
151+
CARGO_INCREMENTAL: '0'
152+
LLVM_PROFILE_FILE: ${{ steps.vars.outputs.LLVM_PROFILE_FILE }}
153+
RUSTC_WRAPPER: ''
154+
RUSTFLAGS: '-C instrument-coverage'
155+
RUSTDOCFLAGS: ''
156+
run: |
157+
## Test
158+
cargo test ${{ steps.vars.outputs.CARGO_FEATURES_OPTION }} --examples --no-fail-fast
139159
- name: Generate coverage data (via `grcov`)
140160
id: coverage
141161
shell: bash
142162
run: |
143163
## Generate coverage data
144-
COVERAGE_REPORT_DIR="target/debug"
164+
outputs() { step_id="${{ github.action }}"; for var in "$@" ; do echo steps.${step_id}.outputs.${var}="${!var}"; echo "${var}=${!var}" >> $GITHUB_OUTPUT; done; }
165+
#
166+
COVERAGE_REPORT_DIR='${{ steps.vars.outputs.COVERAGE_REPORT_DIR }}'
167+
LLVM_PROFILE_FILE='${{ steps.vars.outputs.LLVM_PROFILE_FILE }}'
168+
PROFILES_DIR='${{ steps.vars.outputs.PROFILES_DIR }}'
169+
BINARY_PATH='target/debug'
145170
COVERAGE_REPORT_FILE="${COVERAGE_REPORT_DIR}/lcov.info"
146-
mkdir -p "${COVERAGE_REPORT_DIR}"
147171
# display coverage files
148-
grcov . --output-type files --ignore build.rs --ignore "vendor/*" --ignore "/*" --ignore "[a-zA-Z]:/*" --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()" | sort --unique
172+
grcov . --binary-path "${BINARY_PATH}" --source-dir . --output-type files --ignore build.rs --ignore "examples/*" --ignore "vendor/*" --ignore "/*" --ignore "[a-zA-Z]:/*" --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()" | sort --unique
149173
# generate coverage report
150-
grcov . --output-type lcov --output-path "${COVERAGE_REPORT_FILE}" --branch --ignore build.rs --ignore "vendor/*" --ignore "/*" --ignore "[a-zA-Z]:/*" --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()"
151-
echo "report=${COVERAGE_REPORT_FILE}" >> $GITHUB_OUTPUT
174+
grcov . --binary-path "${BINARY_PATH}" --source-dir . --output-type lcov --output-path "${COVERAGE_REPORT_FILE}" --ignore build.rs --ignore "examples/*" --ignore "vendor/*" --ignore "/*" --ignore "[a-zA-Z]:/*" --excl-br-line "^\s*((debug_)?assert(_eq|_ne)?!|#\[derive\()"
175+
report=${COVERAGE_REPORT_FILE}
176+
outputs report
152177
- name: Upload coverage results (to Codecov.io)
153178
uses: codecov/codecov-action@v3
154179
# if: steps.vars.outputs.HAS_CODECOV_TOKEN

0 commit comments

Comments
 (0)