bug: forge coverage doesn't respect filters when sparse mode is active

[anajuliabit]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (31c24b0 2024-11-14T00:20:56.222819000Z

What command(s) is the bug in?

forge coverage

Operating System

macOS (Apple Silicon)

Describe the bug

foundry.toml

...
sparse_mode = true # compiles files that match filter criteria

┌─[anajulia@MacBook]──[09:52]──[/Users/anajulia/code/work/moonwell-contracts-v2]
└─>:settings/compiler* λ forge coverage --skip script --out artifacts/coverage --mc xWELLUnitTest
[⠊] Compiling...
[⠒] Compiling 328 files with Solc 0.8.19
[⠢] Compiling 12 files with Solc 0.8.10
[⠔] Compiling 18 files with Solc 0.6.12
[⠑] Solc 0.8.10 finished in 229.39ms
[⠃] Solc 0.6.12 finished in 304.15ms

the same filter when running forge test

┌─[anajulia@MacBook]──[09:47]──[/Users/anajulia/code/work/moonwell-contracts-v2]
└─>:D:origin*? λ forge test --skip script --out artifacts/coverage --mc xWELLUnitTest
[⠊] Compiling...
[⠒] Compiling 200 files with Solc 0.8.19
[⠑] Compiling 12 files with Solc 0.8.10
[⠢] Solc 0.8.10 finished in 465.81ms
i```

[grandizzy]

hey! filters are applied, what you see there is the initial phase when the entire project is built for analyzing contracts, for example:

forge coverage --mt test_Increment --skip script
[⠒] Compiling...
[⠃] Compiling 25 files with Solc 0.8.28
[⠘] Solc 0.8.28 finished in 3.72s
Compiler run successful!
Analysing contracts...
Running tests...

Ran 1 test for test/Counter.t.sol:CounterTest
[PASS] test_Increment() (gas: 31874)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 5.58ms (1.47ms CPU time)

Ran 1 test suite in 42.90ms (5.58ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)
| File            | % Lines       | % Statements  | % Branches    | % Funcs       |
|-----------------|---------------|---------------|---------------|---------------|
| src/Counter.sol | 100.00% (2/2) | 100.00% (2/2) | 100.00% (0/0) | 100.00% (2/2) |
| Total           | 100.00% (2/2) | 100.00% (2/2) | 100.00% (0/0) | 100.00% (2/2) |

while without any test match

forge coverage --skip script
[⠒] Compiling...
[⠃] Compiling 25 files with Solc 0.8.28
[⠃] Solc 0.8.28 finished in 3.78s
Compiler run successful!
Analysing contracts...
Running tests...

Ran 2 tests for test/Counter.t.sol:CounterTest
[PASS] testFuzz_SetNumber(uint256) (runs: 256, μ: 32074, ~: 32385)
[PASS] test_Increment() (gas: 31874)
Suite result: ok. 2 passed; 0 failed; 0 skipped; finished in 404.82ms (402.47ms CPU time)

Ran 1 test suite in 407.08ms (404.82ms CPU time): 2 tests passed, 0 failed, 0 skipped (2 total tests)
| File            | % Lines       | % Statements  | % Branches    | % Funcs       |
|-----------------|---------------|---------------|---------------|---------------|
| src/Counter.sol | 100.00% (2/2) | 100.00% (2/2) | 100.00% (0/0) | 100.00% (2/2) |
| Total           | 100.00% (2/2) | 100.00% (2/2) | 100.00% (0/0) | 100.00% (2/2) |

We will be able to make coverage better when porting to solar, see #9317

[zerosnacks]

Possibly useful here if you want to filter the results: forge coverage --no-match-coverage "(script|test|Foo|Bar)", with the caveat:

foundry/crates/evm/coverage/src/lib.rs

Lines 153 to 157 in fbee819

	/// Removes all the coverage items that should be ignored by the filter.
	///
	/// This function should only be called after all the sources were used, otherwise, the output
	/// will be missing the ones that are dependent on them.
	pub fn filter_out_ignored_sources(&mut self, filter: impl Fn(&Path) -> bool) {

sparse_mode is experimental, especially in the context of coverage

[anajuliabit]

Does this mean that there isn’t a way to skip the compilation of certain files using filters when working with coverage? When I use the skip option, it reduces the number of files being compiled, but using --match-contract does not have the same effect. Just to clarify, I am referring to applying the filters to the compilation process, not the test execution.

[zerosnacks]

Correct, the filter applied in coverage can't be done earlier because there may be files dependent on the ignored ones, which will not be covered if we remove the ignored files earlier. Using --skip here would not work as expected for that reason, --no-match-coverage exists to get a filtered view after generating the analysis is done.

For now marking as not planned, it is possible that Solar enables this post V1.