-
Notifications
You must be signed in to change notification settings - Fork 12.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optionally extend branch coverage to instrument the last operand of lazy boolean expressions #124120
Comments
@rustbot label +A-code-coverage |
There's a similar concern for some other boolean expressions that don't directly participate in a branch. For example: // Non-lazy logical operators:
a | b
a & b
// Branch hidden in a standard-library function:
b.then_some(x) So any mode that tries to instrument boolean expressions probably needs to consider whether/how to handle cases like these, too. |
When I originally designed But with branch coverage, I think what we're seeing is closer to an enum-valued setting with multiple levels, e.g.:
That might relieve the awkwardness of having so many “independent” flags that actually do depend on each other. |
Based on #124217 we could accept it in mcdc first as they are required on the same occasion mostly. |
I am unsure about how this should be implemented.
Solution 1. is probably the simplest to implement, at the cost of small runtime bloat. I would rather go with solution 1, because again, slowdown is expected when instrumenting for code coverage |
Yes, we definitely need a better way to organize all the options. Maybe we could use |
…errors coverage: Replace boolean options with a `CoverageLevel` enum After rust-lang#123409, and some discussion at rust-lang#79649 (comment) and rust-lang#124120, it became clear to me that we should have a unified concept of “coverage level”, instead of having several separate boolean flags that aren't actually independent. This PR therefore introduces a `CoverageLevel` enum, to replace the existing boolean flags for `branch` and `mcdc`. The `no-branch` value (for `-Zcoverage-options`) has been renamed to `block`, instructing the compiler to only instrument for block coverage, with no branch coverage or MD/DC instrumentation. `@rustbot` label +A-code-coverage cc `@ZhuUx` `@Lambdaris` `@RenjiSann`
I still don't understand the motivation for wanting to instrument the RHS of lazy boolean operators specifically, but then not also instrument other boolean-valued expressions. Because if instrumenting all booleans is the goal, then I don't think focusing on lazy booleans specifically is a good intermediate step. |
Condition coverage would really only be meant as a building block for MC/DC. The MC/DC criterion only makes sense for boolean expressions with at least two operands, so tracking branch coverage for simple boolean expressions isn't really relevant. if (a && b && c) { ... } We could rewrite this as ` let tmp1 = a && b;
let tmp2 = tmp1 && c;
if (tmp2) { ... } Then, by only testing with The reason why we don't want to instrument all booleans (including simple assignements like For example: fn foo(a: bool) {
let x = a; // 2. check
if x { // 3. check
...
}
}
fn main() {
let res = foo(true); // 1. check
} In this example. if we were to instrument all boolean values, we would end up with 3 tests for the exact same value, when the 3rd one would be enough. Regarding the instrumentation of short-circuit operators only, there are seemingly 2 reasons for that:
|
I've been thinking some more about whether we should have a separate My main concern is that I want to avoid a proliferation of modes that complicate and weaken testing, especially if they'll be contentious to remove later on. On the other hand, I do recognise the implementation benefits of having a separate tier on a temporary basis, so that it can be developed and tested separately from the full complexity of MC/DC, without affecting the main So I'd like to suggest that we primarily think of the Does that sound OK? |
Yes that sounds good. No need to keep condition AND branch coverage available to the user in the long term, as it will likely create confusion. |
coverage: Optionally instrument the RHS of lazy logical operators (This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.) When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch. That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators. --- As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work. `@rustbot` label +A-code-coverage
coverage: Optionally instrument the RHS of lazy logical operators (This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.) When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch. That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators. --- As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work. ``@rustbot`` label +A-code-coverage
coverage: Optionally instrument the RHS of lazy logical operators (This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.) When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch. That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators. --- As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work. ```@rustbot``` label +A-code-coverage
Rollup merge of rust-lang#125756 - Zalathar:branch-on-bool, r=oli-obk coverage: Optionally instrument the RHS of lazy logical operators (This is an updated version of rust-lang#124644 and rust-lang#124402. Fixes rust-lang#124120.) When `||` or `&&` is used outside of a branching context (such as the condition of an `if`), the rightmost value does not directly influence any branching decision, so branch coverage instrumentation does not treat it as its own true-or-false branch. That is a correct and useful interpretation of “branch coverage”, but might be undesirable in some contexts, as described at rust-lang#124120. This PR therefore adds a new coverage level `-Zcoverage-options=condition` that behaves like branch coverage, but also adds additional branch instrumentation to the right-hand-side of lazy boolean operators. --- As discussed at rust-lang#124120 (comment), this is mainly intended as an intermediate step towards fully-featured MC/DC instrumentation. It's likely that we'll eventually want to remove this coverage level (rather than stabilize it), either because it has been incorporated into MC/DC instrumentation, or because it's getting in the way of future MC/DC work. The main appeal of landing it now is so that work on tracking conditions can proceed concurrently with other MC/DC-related work. ````@rustbot```` label +A-code-coverage
…l, r=nnethercote MCDC Coverage: instrument last boolean RHS operands from condition coverage Fresh PR from rust-lang#124652 -- This PR ensures that the top-level boolean expressions that are not part of the control flow are correctly instrumented thanks to condition coverage. See discussion on rust-lang#124120. Depends on `@Zalathar` 's condition coverage implementation rust-lang#125756.
Rollup merge of rust-lang#125766 - RenjiSann:fresh-mcdc-branch-on-bool, r=nnethercote MCDC Coverage: instrument last boolean RHS operands from condition coverage Fresh PR from rust-lang#124652 -- This PR ensures that the top-level boolean expressions that are not part of the control flow are correctly instrumented thanks to condition coverage. See discussion on rust-lang#124120. Depends on `@Zalathar` 's condition coverage implementation rust-lang#125756.
…hercote MCDC Coverage: instrument last boolean RHS operands from condition coverage Fresh PR from #124652 -- This PR ensures that the top-level boolean expressions that are not part of the control flow are correctly instrumented thanks to condition coverage. See discussion on rust-lang/rust#124120. Depends on `@Zalathar` 's condition coverage implementation #125756.
This issue aims to be a discussion about the implementation of branch coverage, to follow up the discussions in #79649 and #124118.
In the following code example, the current implementation of branch coverage does not instrument the outcome of
b
in theindirect
function, because it has no impact on the control flow of the program and is just assigned tov
.a
, however, creates a conditional evaluation ofb
because of the short circuits semantics and thus creates a branch in the control flow.Even though this instrumentation is "branch coverage" according to the definition, there are several reasons for which one would expect
b
to be instrumented as well :@Zalathar gave pertinent reasons on why the "instrument
b
" behavior might not suit everyone's needs, one being that it implies inserting branches in the MIR that are not here by default. Roughly speaking, this would imply that the MIR of the expression would look like the one oflet x = if a || b { true } else { false };
rather thanlet x = if a { b } else { false };
.In order to give the choice to the user, I think we could implement this behavior under another compilation flags:
-Z coverage-options=condition
.This idea is that condition coverage instruments the outcome of each condition inside boolean expressions:
Basically, this is branch coverage with the special case of
b
being instrumented.In terms of implementation, I think the easiest would be to make
condition
implybranch
coverage, and treat the special case only whencondition
is enabled.The text was updated successfully, but these errors were encountered: