Skip to content

Commit

Permalink
Optional test selector (#112)
Browse files Browse the repository at this point in the history
* refactor(karma) Remove karma runner

* Move karma-runner from stryker repo to stryker-karma-runner
* Make the install-module integration test depend on that module
* Fix issues in PluginLoader:
	* Wrong plugin directory
	* Install only the basename of a module directory

* fix(module-integration-test) Set stryker-karma-runner dependency to not-local

* fix(module-integration-test) Set karma version

* refactor(TestRunnerOrchestrator): Create one test selector for all tests

* refactor(TestRunnerOrchestrator) Rename recordCoverage => initialRun

* Renamed recordCoverage to initialRun, because it is more logical. There might not be a code coverage report.

* feat(NoTestSelector) Initial run in TestRunnerOrchestrator

* Make sure the TestRunnerOrchestrator can run the initial test run without testSelector

* feat(NoTestSelector) Log correct number of tests in Stryker

* feat(NoTestSelector) Make test selector optional for runMutations

* fix(deps): Update stryker-api version

* Needed to include the testSelector config option

* feat(NoTestSelector): Add TestSelectorOrchestrator

* The orchestrator is responsible for choosing the correct testSelector based on the testFramework and testSelector options. Also will log warning and debug messages whenever a test selector is chosen.

* fixt(module-it) Up the version of stryker-api in IT deps

* fix(deps) Update api and karma-runner

* feat(NoTestSelector) Improve warning texts

* feat(NoTestSelector) Use TestSelectorOrchestrator from Stryker

* Make sure we use the configured test selector by using the orchestrator

* refactor(testSelectorOrchestrator) Improve debug log

* feat(NoTestSelector) Add testSelector to stryker args

* docs(readme) Document the use of `testSelector`

* fix(testSelector) Add option for testSelector "null"

* When passing testSelector via command line arguments, it will always be passed as string. So 'null' as string will now also be interpreted as `null`

* fix(dep) Set karma version to 1.0.0

* fix(build): Remove grunt-typings from build
  • Loading branch information
nicojs authored and simondel committed Jul 9, 2016
1 parent e197616 commit 43f34d5
Show file tree
Hide file tree
Showing 11 changed files with 472 additions and 158 deletions.
4 changes: 3 additions & 1 deletion .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,9 @@
"--configFile",
"testResources/sampleProject/stryker.conf.js",
"--logLevel",
"debug"
"debug",
"--testFramework",
"jasmine"
],
"cwd": "${workspaceRoot}",
"runtimeExecutable": null,
Expand Down
1 change: 0 additions & 1 deletion Gruntfile.js
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,6 @@ module.exports = function (grunt) {
});

grunt.loadNpmTasks('grunt-contrib-clean');
grunt.loadNpmTasks('grunt-typings');
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-contrib-watch');
grunt.loadNpmTasks('grunt-mocha-test');
Expand Down
33 changes: 23 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,15 @@ You might recognize this way of working from the karma test runner.
If both the config file and command line options are combined, the command line arguments will overrule the options in the config file.

All options are optional except the `files` (or `-f`), `mutate` (or `-m`), `testFramework` and `testRunner` options.
With `files` you configure all files needed to run the tests, except the test framework files themselves (jasmine).

* With `files` you configure all files needed to run the tests, except the test framework files themselves (jasmine).
The order in this list is important, because that will be the order in which the files are loaded.
With `mutate` you configure the subset of files to target for mutation. These should be your source files.
With `testFramework` you configure which test framework you used for your tests. Currently **only** `'jasmine'` is supported.
With `testRunner` you configure which framework should run your tests. Currently **only** `'karma'` is supported.
* With `mutate` you configure the subset of files to target for mutation. These should be your source files.
* With `testRunner` you configure which test runner should run your tests. Currently **only** `'karma'` is supported.
* With `testFramework` you configure which test framework you used for your tests. The value you configure here is passed through to the test runner,
so which values are supporterd here are determined `'jasmine'` and `null` are supported.
* With `testSelector` you configure which test selector should be used for your tests. If this value is left out, the value of the `testFramework` is used
to determine the `testSelector`. Currently **only** `'jasmine'` is supported. As `Stryker` can run without a `testSelector`, you can explicitly disable it by setting the value to `null`.

Both the `files` and `mutate` options are a list of globbing expressions. The globbing expressions will be resolved
using [node glob](https://github.com/isaacs/node-glob). This is the same globbing format you might know from
Expand Down Expand Up @@ -79,19 +83,28 @@ These include: test files, library files, source files (the files selected with
The order of the files specified here will be the order used to load the file in the test runner (karma).
**Example:** `-f node_modules/a-lib/\*\*/\*.js,src/\*\*/\*.js,a.js,test/\*\*/\*.js`

#### Test runner
**Full notation:** `--testRunner`
**Config file key:** `testRunner`
**Description:**
The test runner you want to use. Currently supported runners: `'karma'`
**Example:** `--testFramework 'karma'`

#### Test framework
**Full notation:** `--testFramework`
**Config file key:** `testFramework`
**Description:**
The test framework you want to use. Currently supported frameworks: `'jasmine'`
**Example:** `--testFramework 'jasmine'`

#### Test runner
**Full notation:** `--testRunner`
**Config file key:** `testRunner`
**Description:**
The test runner you want to use. Currently supported runners: `'karma'`
**Example:** `--testFramework 'karma'`
#### Test selector
**Full notation:** `--testSelector`
**Config file key:** `testSelector`
**Description**:
Stryker kan use a test selector to select individual or groups of tests. If a test selector is used, it can potentially speed up the tests,
because only the tests covering a particular mutation are ran. If this value is left out, the value of the `testFramework` is used
to determine the `testSelector`. Currently **only** `'jasmine'` is supported. If you use an other test framework, or you want to disable test selection for an other reason,
you can explicitly disable the testSelector by setting the value to `null`.

#### Log level
**Short notation:** (none)
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@
"grunt-ts": "5.4.0",
"istanbul": "^0.4.0",
"jasmine-core": "^2.4.1",
"karma": "^1.0.0",
"karma": "1.0.0",
"karma-coverage": "^1.0.0",
"karma-jasmine": "^1.0.2",
"karma-phantomjs-launcher": "^1.0.1",
Expand Down
34 changes: 25 additions & 9 deletions src/Stryker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import TestRunnerOrchestrator from './TestRunnerOrchestrator';
import ReporterOrchestrator from './ReporterOrchestrator';
import './jasmine_test_selector/JasmineTestSelector';
import {RunResult, TestResult} from 'stryker-api/test_runner';
import TestSelectorOrchestrator from './TestSelectorOrchestrator';
import MutantRunResultMatcher from './MutantRunResultMatcher';
import InputFileResolver from './InputFileResolver';
import ConfigReader, {CONFIG_SYNTAX_HELP} from './ConfigReader';
Expand Down Expand Up @@ -48,20 +49,20 @@ export default class Stryker {
*/
runMutationTest(): Promise<MutantResult[]> {
let reporter = new ReporterOrchestrator(this.config).createBroadcastReporter();

let testSelector = new TestSelectorOrchestrator(this.config).determineTestSelector();

return new InputFileResolver(this.config.mutate, this.config.files).resolve()
.then(inputFiles => {
let testRunnerOrchestrator = new TestRunnerOrchestrator(this.config, inputFiles, reporter);
return testRunnerOrchestrator.recordCoverage().then(runResults => ({ runResults, inputFiles, testRunnerOrchestrator }))
let testRunnerOrchestrator = new TestRunnerOrchestrator(this.config, inputFiles, testSelector, reporter);
return testRunnerOrchestrator.initialRun().then(runResults => ({ runResults, inputFiles, testRunnerOrchestrator }))
})
.then(tuple => {
let runResults = tuple.runResults;
let inputFiles = tuple.inputFiles;
let testRunnerOrchestrator = tuple.testRunnerOrchestrator;
let unsuccessfulTests = runResults.filter((runResult: RunResult) => !(runResult.failed === 0 && runResult.result === TestResult.Complete));
let unsuccessfulTests = this.filterOutUnsuccesfulResults(runResults);
if (unsuccessfulTests.length === 0) {
log.info(`Initial test run succeeded. Ran ${runResults.length} tests.`);

this.logInitialTestRunSucceeded(runResults);
let mutatorOrchestrator = new MutatorOrchestrator(reporter);
let mutants = mutatorOrchestrator.generateMutants(inputFiles
.filter(inputFile => inputFile.shouldMutate)
Expand All @@ -78,14 +79,18 @@ export default class Stryker {
}
}).then(mutantResults => {
let maybePromise = reporter.wrapUp();
if(isPromise(maybePromise)){
if (isPromise(maybePromise)) {
return maybePromise.then(() => mutantResults);
}else{
} else {
return mutantResults;
}
}
});
}

filterOutUnsuccesfulResults(runResults: RunResult[]) {
return runResults.filter((runResult: RunResult) => !(!runResult.failed && runResult.result === TestResult.Complete));
}

private loadPlugins() {
if (this.config.plugins) {
new PluginLoader(this.config.plugins).load();
Expand All @@ -105,6 +110,16 @@ export default class Stryker {
}
}

private logInitialTestRunSucceeded(runResults: RunResult[]) {
let totalAmountOfTests = 0;
runResults.forEach(result => {
if (result.succeeded) {
totalAmountOfTests += result.succeeded;
}
});
log.info('Initial test run succeeded. Ran %s tests.', totalAmountOfTests);
}

private setGlobalLogLevel() {
log4js.setGlobalLogLevel(this.config.logLevel);
}
Expand Down Expand Up @@ -156,6 +171,7 @@ export default class Stryker {
Example: node_modules/a-lib/**/*.js,src/**/*.js,a.js,test/**/*.js`, list)
.option('--testFramework <name>', `The name of the test framework you want to use`)
.option('--testRunner <name>', `The name of the test runner you want to use`)
.option('--testSelector <name>', `The name of the test selector you want to use`)
.option('-c, --configFile <configFileLocation>', 'A location to a config file. That file should export a function which accepts a "config" object\n' +
CONFIG_SYNTAX_HELP)
.option('--logLevel <level>', 'Set the log4js loglevel. Possible values: fatal, error, warn, info, debug, trace, all and off. Default is "info"')
Expand Down
78 changes: 48 additions & 30 deletions src/TestRunnerOrchestrator.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import {StrykerOptions, InputFile} from 'stryker-api/core';
import {RunResult, RunnerOptions, TestResult} from 'stryker-api/test_runner';
import {TestSelector, TestSelectorFactory} from 'stryker-api/test_selector';
import {TestSelector} from 'stryker-api/test_selector';
import StrykerTempFolder from './utils/StrykerTempFolder';
import IsolatedTestRunnerAdapter from './isolated-runner/IsolatedTestRunnerAdapter';
import IsolatedTestRunnerAdapterFactory from './isolated-runner/IsolatedTestRunnerAdapterFactory';
Expand All @@ -22,23 +22,36 @@ interface FileMap {
interface TestRunnerSandbox {
index: number;
runnerAdapter: IsolatedTestRunnerAdapter;
selector: TestSelector;
fileMap: FileMap;
testSelectionFilePath: string;
}

export default class TestRunnerOrchestrator {

constructor(private options: StrykerOptions, private files: InputFile[], private reporter: Reporter) {
constructor(private options: StrykerOptions, private files: InputFile[], private testSelector: TestSelector, private reporter: Reporter) {
}

recordCoverage(): Promise<RunResult[]> {
let selector = TestSelectorFactory.instance().create(this.options.testFramework, { options: this.options });
initialRun(): Promise<RunResult[]> {
if (this.testSelector) {
return this.initialRunWithTestSelector();
} else {
return this.initalRunWithoutTestSelector();
}
}

private initalRunWithoutTestSelector() {
let testRunner = this.createTestRunner(this.files, true);
return testRunner.run({ timeout: 10000 }).then(testResults => {
testRunner.dispose();
return [testResults];
});
}

private initialRunWithTestSelector() {
let testSelectionFilePath = this.createTestSelectorFileName(this.createTempFolder());
let runnerAdapter = IsolatedTestRunnerAdapterFactory.create(this.createTestRunSettings(this.files, testSelectionFilePath, 0, true));
let runnerAdapter = this.createTestRunner(this.files, true, testSelectionFilePath);
let sandbox: TestRunnerSandbox = {
runnerAdapter,
selector,
fileMap: null,
testSelectionFilePath,
index: 0
Expand All @@ -61,7 +74,7 @@ export default class TestRunnerOrchestrator {
if (mutant.scopedTestIds.length > 0) {
let sandbox = sandboxes.pop();
let sourceFileCopy = sandbox.fileMap[mutant.filename];
return Promise.all([mutant.save(sourceFileCopy), this.selectTests(sandbox, mutant.scopedTestIds)])
return Promise.all([mutant.save(sourceFileCopy), this.selectTestsIfPossible(sandbox, mutant.scopedTestIds)])
.then(() => sandbox.runnerAdapter.run({ timeout: this.calculateTimeout(mutant.timeSpentScopedTests) }))
.then((runResult) => {
let result = this.collectFrozenMutantResult(mutant, runResult);
Expand Down Expand Up @@ -135,7 +148,7 @@ export default class TestRunnerOrchestrator {
: Promise<RunResult[]> {

return new Promise<RunResult[]>(resolve => {
this.selectTests(sandbox, [currentTestIndex])
this.selectTestsIfPossible(sandbox, [currentTestIndex])
.then(() => sandbox.runnerAdapter.run({ timeout: 10000 }))
.then(runResult => {
if (runResult.result === TestResult.Complete && runResult.succeeded > 0 || runResult.failed > 0) {
Expand All @@ -154,36 +167,38 @@ export default class TestRunnerOrchestrator {
}

private createTestRunnerSandboxes(): Promise<TestRunnerSandbox[]> {

return new Promise<TestRunnerSandbox[]>((resolve, reject) => {
let cpuCount = os.cpus().length;
let testRunnerSandboxes: TestRunnerSandbox[] = [];
let allPromises: Promise<any>[] = [];
log.info(`Creating ${cpuCount} test runners (based on cpu count)`);
for (let i = 0; i < cpuCount; i++) {
allPromises.push(this.createSandbox(i).then(sandbox => testRunnerSandboxes.push(sandbox)));
}
Promise.all(allPromises).then(() => resolve(testRunnerSandboxes));
});
let cpuCount = os.cpus().length;
let testRunnerSandboxes: TestRunnerSandbox[] = [];
let allPromises: Promise<any>[] = [];
log.info(`Creating ${cpuCount} test runners (based on cpu count)`);
for (let i = 0; i < cpuCount; i++) {
allPromises.push(this.createSandbox(i).then(sandbox => testRunnerSandboxes.push(sandbox)));
}
return Promise.all(allPromises).then(() => testRunnerSandboxes);
}

private selectTests(sandbox: TestRunnerSandbox, ids: number[]) {
let fileContent = sandbox.selector.select(ids);
return StrykerTempFolder.writeFile(sandbox.testSelectionFilePath, fileContent);
private selectTestsIfPossible(sandbox: TestRunnerSandbox, ids: number[]): Promise<void> {
if (this.testSelector) {
let fileContent = this.testSelector.select(ids);
return StrykerTempFolder.writeFile(sandbox.testSelectionFilePath, fileContent);
}else{
return Promise.resolve(void 0);
}
}

private createSandbox(index: number): Promise<TestRunnerSandbox> {
var tempFolder = this.createTempFolder();
return this.copyAllFilesToFolder(tempFolder).then(fileMap => {
let selector = TestSelectorFactory.instance().create(this.options.testFramework, { options: this.options });
let runnerFiles: InputFile[] = [];
let testSelectionFilePath = this.createTestSelectorFileName(tempFolder);
let testSelectionFilePath: string = null;
if(this.testSelector){
testSelectionFilePath = this.createTestSelectorFileName(tempFolder);
}
this.files.forEach(originalFile => runnerFiles.push({ path: fileMap[originalFile.path], shouldMutate: originalFile.shouldMutate }));
return {
index,
fileMap,
runnerAdapter: IsolatedTestRunnerAdapterFactory.create(this.createTestRunSettings(runnerFiles, testSelectionFilePath, index, false)),
selector,
runnerAdapter: this.createTestRunner(runnerFiles, false, testSelectionFilePath, index),
testSelectionFilePath
};
});
Expand Down Expand Up @@ -214,14 +229,17 @@ export default class TestRunnerOrchestrator {
});
}

private createTestRunSettings(files: InputFile[], testSelectionFilePath: string, index: number, coverageEnabled: boolean): RunnerOptions {
private createTestRunner(files: InputFile[], coverageEnabled: boolean, testSelectionFilePath?: string, index: number = 0): IsolatedTestRunnerAdapter {
if (testSelectionFilePath) {
files = [{ path: testSelectionFilePath, shouldMutate: false }].concat(files);
}
let settings = {
coverageEnabled,
files: [{ path: testSelectionFilePath, shouldMutate: false } ].concat(files),
files,
strykerOptions: this.options,
port: this.options.port + index
};
log.debug(`Creating test runner %s using settings {port: %s, coverageEnabled: %s}`, index, settings.port, settings.coverageEnabled);
return settings;
return IsolatedTestRunnerAdapterFactory.create(settings);
}
}
64 changes: 64 additions & 0 deletions src/TestSelectorOrchestrator.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import {TestSelectorFactory, TestSelector} from 'stryker-api/test_selector';
import {StrykerOptions} from 'stryker-api/core';
import * as log4js from 'log4js';

const WARNING_RUNNING_WITHOUT_SELECTOR = 'Stryker will continue without the ability to select individual tests, thus running all test for every generated mutant.';
const IGNORE_WARNING = 'Set `testSelector` option explicitly to `null` to ignore this warning.';
const log = log4js.getLogger('TestSelectorOrchestrator');

export default class TestSelectorOrchestrator {

constructor(private options: StrykerOptions) {
}

determineTestSelector(): TestSelector {
let testSelector: TestSelector = null;
if (this.options.testSelector && this.options.testSelector !== 'null') {
testSelector = this.determineTestSelectorBasedOnTestSelectorSetting();
} else if (this.options.testSelector === null || this.options.testSelector === 'null') {
log.debug('Running without testSelector (testSelector was null).');
} else {
if (this.options.testFramework) {
testSelector = this.determineTestSelectorBasedOnTestFrameworkSetting();
} else {
log.warn(`Missing config settings \`testFramework\` or \`testSelector\`. ${WARNING_RUNNING_WITHOUT_SELECTOR} ${IGNORE_WARNING}`);
}
}
return testSelector;
}

private determineTestSelectorBasedOnTestSelectorSetting(): TestSelector {
if (this.testSelectorExists(this.options.testSelector)) {
log.debug(`Using testSelector ${this.options.testSelector} based on \`testSelector\` setting`);
return this.createTestSelector(this.options.testSelector);
} else {
log.warn(`Could not find test selector \`${this.options.testSelector}\`. ${WARNING_RUNNING_WITHOUT_SELECTOR} ${this.informAboutKnownTestSelectors()}`);
return null;
}
}

private determineTestSelectorBasedOnTestFrameworkSetting(): TestSelector {
if (this.testSelectorExists(this.options.testFramework)) {
log.debug(`Using testSelector ${this.options.testFramework} based on \`testFramework\` setting`);
return this.createTestSelector(this.options.testFramework);
} else {
log.warn(`Could not find test selector \`${this.options.testFramework}\` (based on the configured testFramework). ${WARNING_RUNNING_WITHOUT_SELECTOR} ${IGNORE_WARNING} ${this.informAboutKnownTestSelectors()}`);
return null;
}
}

private informAboutKnownTestSelectors() {
return `Did you forget to load a plugin? Known test selectors: ${JSON.stringify(TestSelectorFactory.instance().knownNames())}.`;
}

private createTestSelector(name: string) {
return TestSelectorFactory.instance().create(name, this.createSettings());
}
private testSelectorExists(maybeSelector: string) {
return TestSelectorFactory.instance().knownNames().indexOf(maybeSelector) > -1;
}

private createSettings() {
return { options: this.options };
}
}
Loading

0 comments on commit 43f34d5

Please sign in to comment.