Support mutation testing with angular cli

[nicojs]

We're investigating the possibility of supporting angular in Stryker, a mutation testing framework. I was wondering if it was possible to run test with ng test --watch=false --single-run=false and keep open the karma server so we can fire karma run commands.

The idea would be that we make small code changes and rerun the tests with karma run. Ideally we would be able to distinguish between a failed test and a compile error.

[filipesilva]

I read about what you guys are doing and think it's pretty cool!

I'm not familiar with what karma run does. Can you fill me in?

I think it should be possible to just leave the server open. I mean, you can do that right now by simply running ng test (default watch mode). It will stay open, but reload on code changes.

Ultimately it's karma itself that closes the server so this is mostly a matter of configuring karma itself to stay open.

[nicojs]

@filipesilva thanks for your interest and kind words

I had to look it up myself as well. With karma run you can send a command to the karma server to signal a new test run to start.

This is only part of the solution though, as we also need exact test results (inc code coverage results per test if we can). We manage this with karma because we use karma's pubic api (with a custom coverage reporter) to start the karma server and report the results back to us.

To summarize: we're looking for this kind of api:

import * as ng from 'angular-cli';

// Start a test runner. This in turn should load webpack and other plugins karma needs to run the tests
const testRunner = ng.test({ 
  baseDir: 'sandbox/directory/for/this/testRunner',  
  karmaConfig: { /*override karma config*/ plugins: [{  
      'reporter:rawCoverage': ['type', RawCoverageReporter] 
    }, 
    reporters: [ 'rawCoverage' ]
  }
});

// Forward karma's events so we can collect the data we need
testRunner.on('spec_complete', (browser, spec) => {} ); 
testRunner.on('run_complete', (browser, runResult) => {} ); 
testRunner.on('browser_error', (browser, error) => {} ); 
testRunner.on('raw_coverage_complete', (browser, error) => {} ); // => a custom event that we report from the RawCoverageReporter

 // A way to signal we want a new test run to start
testRunner.run();

That way we can write mutated typescript code to the sandbox directory (directory where we keep a copy of your source code), and signal a new test run (we run multiple sandboxes in parallel for performance reasons).

I think it should be possible to just leave the server open. I mean, you can do that right now by simply running ng test (default watch mode). It will stay open, but reload on code changes.

True, however it doesn't seem to be possible to set singleRun: false in combination with watch: false. Also, this would be only part of the solution as stated above.

Another way of working would be to do all of the work the angular cli is doing ourselves. @Archcry did an awesome proof of concept project here: https://github.com/Archcry/stryker-angular-extra. Here, he basically mimics the angular cli build functionality using the stryker-webpack-transpiler in combination with the stryker-karma-runner. The problem with this approach is that it duplicates the angular-cli test functionality in Stryker, thus i feel it is not maintainable in the long run. However, it might be a sollution if there is a way to pull the entire webpack configuration from angular-cli at runtime (I don't know if that is even possible, with the setup angular-cli is using with regards to webpack and karma).

I hope this clears for you. Don't hesitate to ask more questions. We also have a technical reference page on our website.

Thanks for the help!

[nicojs]

I had some time to think about it in the hope of finding something less intrusive for angular-cli and yet maintainable for us (the Stryker team). I might have a solution.

Would it be possible to use @angular/cli to create a webpack compiler object for us? It would basically look like an ng test with some minor tweaks:

• It should also build the tests (in a separate bundle or not doesn't really matter)
• It should give us the possibility to override the input and output file systems, @Archry's stryker-webpack-transpiler plugin does here: https://github.com/Archcry/stryker-webpack/blob/master/src/compiler/WebpackCompiler.ts#L38
• It should result in a webpack compiler object, so we can could call compiler.run on it when we're mutating the in-memory files.

@filipesilva do you think that's possible? If you give us some pointers, maybe we can help with a PR on your side?

[nicojs]

Also see:
#9089

@filipesilva you've stated about https://angular.io/guide/webpack:

I'm sorry to be the bearer of bad news, but that guide is old and I think is deprecated. I doesn't appear on the sidebar anymore I think.

What does that mean exactly? Is it deprecated because it might break in the future based on current development? Or is it deprecated because you're not maintaining the docs anymore?

We would like to use this approach as we know it works, but we wouldn't want to build a solution based on something that we know we cannot support.

[filipesilva]

I mean that specific guide (the webpack one) is deprecated and not being maintained anymore. I suppose it still works but it's been a while since it was updated. The page is still live but not linked to from anywhere IIRC. We do not maintain the plugins used in that example either.

[nicojs]

@filipesilva thanks for your feedback.

Just one more question. You also stated this yesterday:

Well, the Angular Compiler proper cares about TS files and nothing else. It asks the file system, or a given resource loader, for html/css/sass/etc.

I believe this is the case for the awesome-typescript-loader as well. Does that mean that the awesome-typescript-loader can be a drop-in replacement for the AngularCompilerPlugin (without aot and maybe other performance improvements)?

EDIT:

It sure looks like it is. If I add this line in my webpack.config everything seems to work:

webpackConfig.module.rules.forEach(rule => {
  if (rule.loader === '@ngtools/webpack') {
    delete rule.loader;
    rule.loaders = [
      {
        loader: 'awesome-typescript-loader',
        options: {
          configFileName: path.join(process.cwd(), 'src', appConfig.testTsconfig)
        }
      }, 'angular2-template-loader'
    ]
  }
});

[filipesilva]

awesome-typescript-loader is a TS only compiler, and can be used for JIT only compilation. Only AOT compilation needs @ngtools/webpack. Currently unit tests can only be run in JIT mode anyway, but there will be support for AOT unit testing in the future.

[nicojs]

Cool! This approach seems to be working. I will create an example which uses private angular cli apis to build a webpack config. Once I have that, I will propose a public api here.

@filipesilva thanks for all your afford up until now. 💐

[filipesilva]

No problem at all. I think what you all are doing is super cool 😄

Sorry it wasn't more straightforward to integrate.

[nicojs]

@filipesilva
We are in the middle of merging our webpack transpiler plugin into master and have a working integration test with an angular cli plugin 😌.

See line 3 to 17 in the webpack config. We use private angular-cli apis to reconstruct the webpack config:
https://github.com/stryker-mutator/stryker/blob/b83e304f2de2bb5a60c22cb22ccafdd08cfcc9ce/integrationTest/test/angular-project/webpack-stryker.config.js#L3-L17

Would it be OK to move all that into a single public api call? That would be great as it would allow users to simplify their webpack config used for Stryker, as well as make sure the API doesn't break between patch and minor releases.

EDIT: I think the api could look something like this:

import { buildTestWebpackConfig, getAppFromConfig } from '@angular/cli';

const appConfig = getAppFromConfig(undefined /* app name */);
const webpackConfig = buildTestWebpackConfig(appConfig );

[nicojs]

@filipesilva stryker-webpack-transpiler has been released! See https://github.com/nicojs/angular-stryker-example where I explain how to use it for Anguler.

As a next step, I would like to work on the changes in the angular cli public api. Could you please share your thoughts my comment above? Thanks!