I am not able to run striker on requirejs project

[sharikovvladislav]

Hi,

I am getting this:

Running "stryker:bob" (stryker) task
[2016-11-05 03:35:50.312] [DEBUG] ConfigReader - Loading config test/units/cfg/stryker.conf.js
[2016-11-05 03:35:51.287] [INFO] InputFileResolver - Found 205 file(s) to be mutated.
[2016-11-05 03:35:51.288] [INFO] TestRunnerOrchestrator - Starting initial test run. This may take a while.
[2016-11-05 03:35:58.250] [ERROR] IsolatedTestRunnerAdapter - (node:13984) DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.

[2016-11-05 03:36:13.653] [ERROR] Stryker - One or more tests errored in the initial test run:
        ReferenceError: Can't find variable: define
at http://localhost:9234/base/User_Part/Source/htdocs/ps/ng-s-cli/common/js/directives/collapseHandler.js?de9fca0db0e7078b727965c2d0e6e5b4de068f6b:9
Fatal error: Stryker was unable to run the mutation test

When I run stryker.

How I can manage this?

[nicojs]

Did you load requirejs in your stryker.conf.js?

How would you load requirejs in your normal karma.conf.js setup? Is it with a plugin? Or is it in the files list?

[sharikovvladislav]

Hm. I am not sure I load requirejs in stryker.conf.js. Here is it:

module.exports = function(config){
  config.set({
    files: [
      // Add your files here, this is just an example:
      { pattern: 'some/path/**/*.js', mutated: true, included: true},
      'test/units/tests/**/*.js'
    ],
    testRunner: 'karma',
    testFramework: 'jasmine',
    reporter: ['html', 'progress'],
    logLevel: 'debug'
  });
};

Sure I run my tests (I am doing it with karma) with requirejs plugin.

The reason why I didn't enable requirejs in stryker is: I didn't find it on this page: https://www.npmjs.com/search?q=stryker-plugin Is requirejs is build in plugin? How can I use it then? How I can enbale requirejs support? Should I just add it to path? As I understand there: #159

[sharikovvladislav]

Oooh ok. I have to use this: https://www.npmjs.com/package/stryker-karma-runner

I am trying this at the moment.

[sharikovvladislav]

1. Can I just add karmaConfig from another file? I have my normal karma.config.js. Can I include it to the property karmaConfig?

2. Also what is 'filesToMutate' property? I found it there: https://www.npmjs.com/package/stryker-karma-runner

3. What should be in stryker.conf.js at main config and what should be in karmaConfig property? Should I add all files

4. Should stryker work above karma-runner and requirejs plugin? I can't get tests running with stryker even on dry test run (without mutations).
   I have this prolem: Uncaught Error: Script error for "ngRoute":

D:\Dev\Sandbox\sbms_s_cli>grunt stryker
sonarRunner:auto = preview

Running "stryker:bob" (stryker) task
[2016-11-06 15:00:53.847] [DEBUG] ConfigReader - Loading config test/units/cfg/stryker.conf.js
[2016-11-06 15:00:53.850] [DEBUG] PluginLoader - Loading stryker-* from D:\Dev\Sandbox\sbms_s_cli\node_modules
[2016-11-06 15:00:53.851] [DEBUG] PluginLoader - Loading plugins stryker-html-reporter (matched with expression stryker-*)
[2016-11-06 15:00:53.851] [DEBUG] PluginLoader - Loading plugins stryker-karma-runner (matched with expression stryker-*)
[2016-11-06 15:00:53.851] [DEBUG] PluginLoader - Loading plugins stryker-html-reporter
[2016-11-06 15:00:53.926] [DEBUG] PluginLoader - Loading plugins stryker-karma-runner
[2016-11-06 15:00:53.956] [DEBUG] Stryker - Using config: {"logLevel":"debug","testFramework":"jasmine","testRunner":"karma","timeoutMs":5000,"timeoutFactor":1.5,"plugins":["stryker-*"],"port":9234,"reporter":["html","progress"],"files":["test/units/tools/ng-core/c
ore-all.js","test/units/tools/ng-components/components-all.js","test/units/main.js",{"pattern":"test/units/tools/ng-core/**/*.*","included":false},{"pattern":"test/units/tools/ng-components/**/*.*","included":false},"User_Part/Source/htdocs/ps/ng-s-cli/define.js","
User_Part/Source/htdocs/ps/ng-s-cli/customer/define.js","User_Part/Source/htdocs/ps/ng-s-cli/credit/js/define.js","User_Part/Source/htdocs/ps/ng-s-cli/reservation/js/define.js",{"pattern":"test/units/tests/**/*.js","included":false},{"pattern":"test/units/tests/cus
tomer/service/**/*.js","included":false},{"pattern":"User_Part/Source/htdocs/ps/ng-s-cli/**/*.*","included":false},{"pattern":"User_Part/Source/htdocs/ps/ng-s-cli/customer/**/*.js","included":false},{"pattern":"User_Part/Source/htdocs/ps/ng-s-cli/customer/js/servic
es/**/*.js","included":false},{"pattern":"test/units/fixtures/*.json","included":false}],"karmaConfig":{"browsers":["Chrome"],"frameworks":["jasmine","requirejs"],"autoWatch":false,"singleRun":false},"configFile":"test/units/cfg/stryker.conf.js"}
[2016-11-06 15:00:53.958] [DEBUG] TestSelectorOrchestrator - Using testSelector jasmine based on `testFramework` setting
[2016-11-06 15:00:54.146] [WARN] InputFileResolver - No files marked to be mutated, stryker will perform a dry-run without actually mutating anything.
[2016-11-06 15:00:54.147] [INFO] TestRunnerOrchestrator - Starting initial test run. This may take a while.
[2016-11-06 15:00:54.147] [DEBUG] TestRunnerOrchestrator - Creating a sandbox for files in D:\Dev\Sandbox\sbms_s_cli\.stryker-tmp\9599672\test-runner-files4109480
[2016-11-06 15:00:54.147] [DEBUG] TestRunnerOrchestrator - Creating test runner 0 using settings {port: 9234, coverageEnabled: true}
[2016-11-06 15:00:56.041] [ERROR] IsolatedTestRunnerAdapter - (node:10440) DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.

[2016-11-06 15:01:00.339] [ERROR] Stryker - One or more tests errored in the initial test run:
        Uncaught Error: Script error for "ngRoute"
http://requirejs.org/docs/errors.html#scripterror
at http://localhost:9234/base/node_modules/requirejs/require.js?640c874a2dac17894676783f02cf5dd1ba4744af:143
Fatal error: Stryker was unable to run the mutation test


Execution Time (2016-11-06 15:00:51 UTC+3)
loading tasks    2s  ███████████████████████████████████████████████████████ 23%
stryker:bob    6.5s  ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 76%
Total 8.5s

I copypasted files from my karma.conf.js. Here is my stryker.conf.js:

module.exports = function(config){
  config.set({
    files: [
      'test/units/tools/ng-core/core-all.js',
      'test/units/tools/ng-components/components-all.js',
      'test/units/main.js',
      {pattern: 'test/units/tools/ng-core/**/*.*', included: false},
      {pattern: 'test/units/tools/ng-components/**/*.*', included: false},
      'User_Part/Source/htdocs/ps/ng-s-cli/define.js',
      'User_Part/Source/htdocs/ps/ng-s-cli/customer/define.js',
      'User_Part/Source/htdocs/ps/ng-s-cli/credit/js/define.js',
      'User_Part/Source/htdocs/ps/ng-s-cli/reservation/js/define.js',
      {pattern: 'test/units/tests/**/*.js', included: false},
      {pattern: 'test/units/tests/customer/service/**/*.js', included: false},
      {pattern: 'User_Part/Source/htdocs/ps/ng-s-cli/**/*.*', included: false},
      {pattern: 'User_Part/Source/htdocs/ps/ng-s-cli/customer/**/*.js', included: false},
      {pattern: 'User_Part/Source/htdocs/ps/ng-s-cli/customer/js/services/**/*.js', included: false},
      {pattern: 'test/units/fixtures/*.json', included: false}
    ],
    testRunner: 'karma',
    testFramework: 'jasmine',
    karmaConfig: { // these are the defaults
      browsers: ['Chrome'],
      frameworks: ['jasmine', 'requirejs'],
      autoWatch: false,
      singleRun: false
    },
    reporter: ['html', 'progress'],
    logLevel: 'debug'
  });
};

My main.js:

(function () {
    var allTestFiles, REGEXP_SPEC, files, match_spec;
    var coreModules = ['ngRoute', 'ngMocks'];
    allTestFiles = [];
    REGEXP_SPEC = /(spec|test)\.js$/i;
    files = [];

    /* globals window: false*/
    Object.keys(window.__karma__.files).forEach(function (file) {
        if (REGEXP_SPEC.test(file)) {
            match_spec = file.match(/\w+\.(spec|test)\.js$/i);
            files.push(match_spec[0].split('.')[0]);
            // Normalize paths to RequireJS module names.//
            allTestFiles.push(file);
        }
    });

    var startTest = function () {
        require(allTestFiles, function () { // указываем тестируемые файлы
            window.__karma__.start();
        });
    };

    require.config({
        baseUrl: 'http://localhost:9876/ps/',
        // dynamically load all test files
        deps: coreModules,
        // we have to kickoff jasmine, as it is asynchronous
        callback: startTest
    });
})();

So ngRoute is core dependency in main.js. But actually it seems to me I don't get into the main.js file.... I put console.log in it and don't see anytying in the output.

[nicojs]

You are on the right track. Indeed, right now its not possible to make stryker use your karma.conf.js file, you should add these options to the karmaConfig property for now as described here
https://www.npmjs.com/package/stryker-karma-runner#override-karma-config

I am on the road right now, i will take a look and respond in a few hours.

[sharikovvladislav]

Ok so I did good. Am I right in this opinion: files to be mutaded (my source files) must be in filesToMutate property, right? Looks like I am on the correct way.

Also look this. This is loading of karma page with stryker:
https://i.gyazo.com/3f38842d8b0ca42a15e8ca4ca6961480.png

And here is without it:
https://i.gyazo.com/5f12818fa2a67d654703554932382bf2.png

In stryker case I am getting error about ngRoutes and ngMocks. ngRoutes and ngMocks are RequireJS modules wich are defined in core-all.js file. This file is not loaded in stryker case thats why I am getting error about ngMocks (or ngRoute - it floats). Why it is not loading? How I can load it before main.js?

[sharikovvladislav]

Looks like stryker and karma parse files property differently. I am talking about this block:

'test/units/tools/ng-core/core-all.js',
'test/units/tools/ng-components/components-all.js',
'test/units/main.js',
{pattern: 'test/units/tools/ng-core/**/*.*', included: false},
{pattern: 'test/units/tools/ng-components/**/*.*', included: false},

I need next line, obviously, to get all these files indexed by karma server (so they are available under /base path.

{pattern: 'test/units/tools/ng-core/**/*.*', included: false},

So karma load core-all.js, add all files to index, define ngRoute and ngMocks modules and when main.js is requiring them everything is ok. If I remove this line I wouldn't be able to bootstrap my core framework (because I need all files under ng-core/ to be available under /base path).

Ok, what is in case of stryker. Looks like stryker parse files option in different way.

If I have this in conf:

'test/units/tools/ng-core/core-all.js',
'test/units/tools/ng-components/components-all.js',
'test/units/main.js',
{pattern: 'test/units/tools/ng-core/**/*.*', included: false},
{pattern: 'test/units/tools/ng-components/**/*.*', included: false},

Then files core-all.js and components are not loaded. Probably this happens because I have included: false on paths 'test/units/tools/ng-core/**/*.*' and 'test/units/tools/ng-components/**/*.*'.

So if I remove this lines I see that core-all.js and components-all.js are loading, but application is not able to bootstrap because there are no mandatory files under /base path (like lib/jquery/*** etc).

What do you think about it?

Also, is there any property like proxies (like in karma) in stryker?

[nicojs]

FilesToMutate was renamed to mutate in the 0.4 release of stryker. Appearantly we forgot to update the stryker-karma-runner docs.

You can either use that property, or use the object literal to mark files to be mutated as described here: https://github.com/stryker-mutator/stryker/blob/master/README.md#all-files

[sharikovvladislav]

OK. I got about mutate property or another way to set up files to mutate.
Looking forward to hearing answer from you about "files" problem.

[nicojs]

I think i know what the issue is. Indeed, it has to do with the file list. Consider these 2 lines:

'test/units/tools/ng-core/core-all.js',
{ pattern: 'test/units/tools/ng-core/**/*.*', included: false },

The second pattern will localize the core-all file again. When using karma directly, it would deduplicate it, letting the first pattern "win". Stryker does not do that.

Can you try to write your patterns in such a way that you won't define a file twice? Just to see if that fixes the issue?

I tried to use requirejs+karma+stryker myself and ran into the same issue. You can see it working here: https://github.com/nicojs/karma-requirejs-stryker

First i had these patterns:

{ pattern: 'relative/**/*.js', mutated: true, included: false },
{ pattern: 'test-main.js', mutated: false, included: true },
{ pattern: '*.js)', mutated: false, included: false },

As you can see, test-main.js would be located twice. Once by the second pattern and once by the third. Then i rewrote it like this:

{ pattern: 'relative/**/*.js', mutated: true, included: false },
{ pattern: 'test-main.js', mutated: false, included: true },
{ pattern: '+(dependency.js|legacy-library.js|test.js)', mutated: false, included: false },

Now it works.

[sharikovvladislav]

I already tried this but I had strange problem. If I remove line {pattern: 'test/units/tools/ng-core/**/*.*', included: false} line, I will get Uncaught Error: Script error for "jquery", needed by: x, y error. Ok lets add jquery paths to the files property. I am adding this:{pattern: 'test/units/tools/ng-core/lib/jquery/*.js',included: false}, and it did not work. Ok I tried this:

{pattern: 'test/units/tools/ng-core/lib/jquery/jquery-2.2.3.min.js',included: false},
{pattern: 'test/units/tools/ng-core/lib/jquery/jquery-selectors.js',included: false},
{pattern: 'test/units/tools/ng-core/lib/jquery/jquery-ie-special-events.js',included: false},

It didnt' work too.

Now I want to debug script execution. I want to see files in window.__karma__.files property in main.js. I put breakpoint in my main.js file. I started debugger and debugger didn't stopped at the breakpoint. Wooot? It looks really strange. I couldn't stop at the breakpoints in main.js, I don't see console.log outputs. What? OK, I removed all lines from main.js file and so? Ok now runner get broken. I tried to change name of ngRoute to ngRouteXXX. I got another error: Uncaught Error: Script error for "ngRouteXXX". So runner loads main.js and trying to work with code in it. But why I can't debug it and put console.log in it?

I need this to check window.karma.files property contents. I need to check if jquery/** files was indexed.

If I debug karma-runner (without stryker) - everything is ok. Do you know why I am not able to debug main.js?

Note that if I put debugger in grunt configuration - debugger stops on this breakpoint.

[sharikovvladislav]

1. I also tried to debug or console.log something in your test project (https://github.com/nicojs/karma-requirejs-stryker) and get nothing. Looks like stryker inejcts to the karma-runner too hard :) And eats everything like console.log. Isn't it?
2. Problem with files. Changing file paths this way looks like workaround. Isn't it? It is problem with stryker itself?

[nicojs]

I'm not sure what your debug procedure is. Stryker currently always runs test runners in a child process, so that might hinder debugging. This is so we can upscale to multiple instances when starting to test actual mutants. You can see the output of the child processes when you set loglevel to trace: logLevel: 'trace'.

I've created #172 for the duplicate files.

I also created stryker-mutator/stryker-karma-runner#7 for the reading of the karma.conf file and related bug #148 to that one as well.

If these 3 bugs are fixed, it might work for you right out of the box :)

We're also working on a major release which would change the way the initial test run is done, see progress here: #165

[sharikovvladislav]

Debug procedure... Hm. I will try to explain :) Imagine you are creating 1-3 line application :) You have document, you add DOMContentLoaded event listener on it. So your code is:

  document.addEventListener("DOMContentLoaded", function(event) {
    debugger;
    console.log("DOM fully loaded and parsed");
  });

Now when you open this page in browser in debug mode (when Developer Tools is opened (or FireBug)) script execution will stop on this breakpoint debugger.

So when I say debug main.js I mean I put debugger keyword in it or setting up breakpoint (like in Chrome DevTools or any other tool which is able to debug JS code).

And when I run karma runner and stryker above it I am not able to debug main.js. This is the problem. I don't know how to explain more correctly. I can make a video :) (I am not trolling or joking sry, I know I have bad english and it is therefore you didn't understand me).

[sharikovvladislav]

A-ha! I just understood what is the problem there.

Karma runner has property proxies. When karma doesn't find field it checks property proxies and parts in it for requested file. If no, you will get 404.

Do you have option like proxies in stryker?

[sharikovvladislav]

Finally!

1. I added property proxies to karmaConf section (and again, this is argument for adding ability to just include karma.conf.js by path (through require).
2. I changed default port to 9876 (or I have to change ports in proxies and in main.js)

Now I am getting all libraries, all tests (main.js) and require all modules they test. Really beautifull. I have another problem: it looks like test page is reloading. When I watch net traffic I see page is reloading right after all modules to test were required.

UPD
Wow! I just tried to remove all tests except one and realised that paged reloaded only about 10 times. I check how much its do i have in my test. There is 11 it`s. I removed all except 3 and got page reloaded 4 times. I removed 1 more and page reloaded 3 times. When I have 1 it page reloads two times. When I have 2000 tests (real value) page will reload 2001 time, obviously.

Do you want what is it?

[nicojs]

@sharikovvladislav very cool you got it working! I'm really happy, its been a fruitful exercise as we now have a few more valuable issues. We will be fixing them in the coming weeks. Maybe you want to verify them as we fix them? The idea is that it should work out-of-the-box in the near future.

Wow! 2000 tests! That's awesome! No, loading the page 2001 times is not what we want, but its how it is implemented right now. For reasons why: see github.com//issues/142#issuecomment-244852473. You can disable this with testSelector: null in your config (this setting will be removed in the next major version of stryker in favor of coverageAnalysis: 'off'). Disabling the test selector makes the initial test run fast, but the mutation testing itself very slow as Stryker will run all tests for all mutants it wants to test.

We'll be releasing the next major version: 0.5 of Stryker somewhere in the next weeks. You can see progress here: #165. That release will make the initial test run collect coverage reports in one pass of the test runner. So 1 page load instead of 2001 in your case.

So please be patient as we fix this issue :)

[sharikovvladislav]

@nicojs Sure I will try to review fixed issues as soon as possible.

About deduplicating files. I think it is really must have feature. I have 12 items in files block instead of 4 lines (like in karma conf file).

About 2001 tests. The problem is not only in 2001 tests. The problem is application is about 100 files and these 100 files is requiring by requirejs each it.

Also what about no debug ability? Do you have ideas how to fix this problem?

I have another problem :) I tried mutation testing on small test case (about 10 its). After I enabled mutants I got 404 error on files which are marked asmutated: true. It looks like stryker mutated marked files and removed them from/base` path of karma server. How I can fix this?

Do you have some small tasks to implement (in stryker)? I can try to help somewhere with them.

[nicojs]

About deduplicating files. I think it is really must have feature. I have 12 items in files block instead of 4 lines (like in karma conf file).

This is #172

About 2001 tests. The problem is not only in 2001 tests. The problem is application is about 100 files and these 100 files is requiring by requirejs each it.

How long does a normal karma run take? We can do optimisations in Stryker. Namely running multiple instances of Karma at once and analysis on code coverage to see which tests we need to run. We'll see how fast it will be. At the very least you'll have the progress reporter to report progress to you.

Also what about no debug ability? Do you have ideas how to fix this problem?

Yes we have: #173

I have another problem :) I tried mutation testing on small test case (about 10 its). After I enabled mutants I got 404 error on files which are marked as mutated: true. It looks like stryker mutated marked files and removed them from/base` path of karma server. How I can fix this?

It should not remove mutated files from the base path. However, after the initial run and in the mutation run, all files are copied to a local stryker-tmp folder and ran from there. Could that be causing troubles?

Do you have some small tasks to implement (in stryker)? I can try to help somewhere with them.

Yes, we have tons :). Maybe you can start with these:

#163
#162

Be sure you can run npm test locally. We've optimised the project for vscode so if your using that editor, you can see we already have tasks defined. During development, you might want to run the typescript compiler in the background (tsc -w)

[simondel]

I'll close this issue for now since we haven't heard anything anymore. If you still have issues, could you reopen it? Thanks!