Beta: Mutations are creating invalid code #2469
Comments
|
Thanks a lot @gramster ! Will have a look tomorrow |
|
Update: I just read #2466 and that triggered me. I think this issue is already fixed, but we didn't release a new beta version. I just released Sorry for the work you put in, we should be better about updating our beta's 😢 It was fixed with #2438 EDIT: I've added some more instructions in #2466 |
|
No worries, I am very excited about getting this to work as it solves a big problem for me around measuring the effectiveness of our test suites :-), and you have been very responsive. I'll update and try again. |
|
Looks like I am now past the build step! The next thing I need to figure out is how to customize the mocha command. |
|
I got somewhat further. I had to remove a bunch of tests that failed in the initial test run, but now I get: and I have no idea what tests I should remove to get past this (there are several thousand tests). |
|
Well... the initial test run should pass if your normal test run also passes. Instrumented code without an activated mutant should of course run the same way as non-instrumented code would. There can be a number of things that are problematic. The best way to debug this is to kill the process during the initial test run, I can also give it a try. Is this the repo? https://github.com/Microsoft/vscode-python EDIT: Found your remarks in the other issue, see that that's the repo :) |
|
Alternatively, you can run with |
|
I've taken a look. Nice bug! 🐛 This: switch (item.severity) {
case DiagnosticSeverity.Error: {
traceError(message);
this.outputChannel.appendLine(message);
break;
}
// ...
default: {
traceInfo(message);
}
}Is instrumented as this: switch (item.severity) {
case DiagnosticSeverity.Error:
{
switch (__global_69fa48.__activeMutant__) {
case 1775:
{}
break;
default:
__global_69fa48.__coverMutant__(1775);
{
traceError(message);
this.outputChannel.appendLine(message);
break;
}
break;
}
}
// ...
default:
{
switch (__global_69fa48.__activeMutant__) {
case 1779:
{}
break;
default:
__global_69fa48.__coverMutant__(1779);
{
traceInfo(message);
}
break;
}
}
}Which... is obviously wrong. The inner @hugo-vrijswijk I know you use the same EDIT: For reference, stryker config I used: {
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"buildCommand": "tsc -b",
"mochaOptions": {
"config": "./build/.mocha.unittests.js.json"
},
"testRunner": "mocha"
} |
@nicojs In Scala you don't have to 'break' from pattern matches, because they are already scoped and there's no fall-through. So there's no issue with a For placing mutations, a single mutation-switch pattern match is created 'above' any other pattern-matching. So this: list.nonEmpty match {
case true => "nonEmpty"
case _ => otherValue
}Would have 3 mutations ( _root_.scala.sys.props.get("ACTIVE_MUTATION") match {
case Some("0") =>
list.isEmpty match {
case true => "nonEmpty"
case _ => otherValue
}
case Some("1") =>
list.nonEmpty match {
case false => "nonEmpty"
case _ => otherValue
}
case Some("2") =>
list.nonEmpty match {
case true => ""
case _ => otherValue
}
case _ =>
list.nonEmpty match {
case true => "nonEmpty"
case _ => otherValue
}
} |
|
Thanks for diving into this, Nico! Really appreciate it. Just BTW, I am wanting to use Stryker because I think our test suite has a lot of anti-patterns in it, especially over-use of mocking, and want to come up with metrics for a baseline that we can improve. I'd be interested in knowing of over metrics you may be aware of. Right now, I am thinking of these:
|
|
@hugo-vrijswijk thanks for pitching in. So scala doesn't have a Since JS does have a break statement, I think the easiest thing to do is to use an
Yeah, I saw some of them already at a first glance (for example
Yep. Definitely "test effectiveness", that's what I call it as well. It might be difficult to get Stryker running on the entire test suite since it takes a very long time. Worst case scenario: Tmutation-testing = Ttest run * n. Why don't you join our slack, so we can discuss the other points further? Always nice to talk about tests 😍 join slack |
Place mutants in a statement with a `if` statement instead of a `switch` statement. Fixes #2469
Place mutants in a statement with an `if` statement instead of a `switch` statement. Fixes #2469
|
Ok after I fixed #2474, I ran into #2485 . After I fixed that I managed to start a run properly: Used this config: {
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"buildCommand": "tsc -b",
"mochaOptions": {
"config": "./build/.mocha.unittests.js.json"
},
"testRunner": "mocha",
"coverageAnalysis": "perTest"
} Hope you have some decent hardware to run this on since it will take a couple of hours. Or run mutation testing on a part of the code. I will release a new version of the beta and report back here. |
|
Just released |
|
Awesome, thanks Nico!
…On Thu, Sep 17, 2020 at 6:06 AM Nico Jansen ***@***.***> wrote:
Just released 4.0.0-beta.7. Happy mutation testing @gramster
<https://github.com/gramster> 😁
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2469 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAVCPCDMLR7BVM6T26MW3ZTSGICWXANCNFSM4RDEHLNA>
.
|
|
And thank you for the bug reports. 🐛 |
Summary
The mutations that are being created have strange errors in them.
Unterminated Template Literals
Below is the tail end of a file are being mutated:
Note that the
\https:` template is unterminated, so when this is compiled by TypeScript compiler you get the error:Unterminated String Literal
Tail end of another file:
It's just truncated at that point, and so the compiler error is:
Original unmutated code:
(There's another 60 lines of code after that I left out that was just dropped from the mutated file).
')' Expected
Another tail end:
Again, the
\https` is unterminated, but that leads to the parentheses being unmatched too, which is the compiler error reported here.Original unmutated code:
There's a bunch more of these but the above are illustrative.
Stryker config
Test runner config
Stryker environment
Test runner environment
# Test commandYour Environment
Add stryker.log
The text was updated successfully, but these errors were encountered: