Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beta: Stryker can get confused by its own reports directory #2466

Closed
gramster opened this issue Sep 9, 2020 · 7 comments
Closed

Beta: Stryker can get confused by its own reports directory #2466

gramster opened this issue Sep 9, 2020 · 7 comments
Labels
🐛 Bug Something isn't working
Milestone

Comments

@gramster
Copy link

gramster commented Sep 9, 2020

Question

I'm making this a question for now but perhaps it should be labelled a bug. I'm using the beta, and worked around the issue #2465 for now, but now after instrumentation nothing happens. I have detailed logging on, and I left it running for hours, but the last thing in the log is:

$ npx stryker run --fileLogLevel trace --logLevel debug
...
21:32:56 (23497) DEBUG Instrumenter Instrumented src/client/workspaceSymbols/parser.ts (120 mutant(s))
21:32:56 (23497) DEBUG Instrumenter Instrumented src/client/workspaceSymbols/provider.ts (33 mutant(s))
21:32:56 (23497) INFO Instrumenter Instrumented 768 source file(s) with 48482 mutant(s)

and nothing happens after that.

Config:

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "packageManager": "npm",
  "reporters": [
    "html",
    "clear-text",
    "progress"
  ],
  "testRunner": "mocha",
  "buildCommand": "npx gulp prePublishNonBundle",
  "mutate": ["src/client/**/*.ts"],
  "coverageAnalysis": "perTest"
}

This is for the Python extension for Visual Studio Code, so it should be reproducable.

It's very possible that my configuration is not right, but I still wouldn't expect a hang like this.

Stryker environment

├─┬ UNMET PEER DEPENDENCY @stryker-mutator/core@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ ├─┬ @stryker-mutator/instrumenter@4.0.0-beta.4
│ │ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ │ ├─┬ @stryker-mutator/util@4.0.0-beta.4
│ ├─┬ @stryker-mutator/util@4.0.0-beta.4
├── @stryker-mutator/javascript-mutator@4.0.0-beta.4
├─┬ @stryker-mutator/mocha-framework@3.3.1
│ ├─┬ @stryker-mutator/api@3.3.1
│ └── @stryker-mutator/util@3.3.1
├─┬ @stryker-mutator/mocha-runner@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ ├─┬ @stryker-mutator/util@4.0.0-beta.4
├── @stryker-mutator/typescript@4.0.0-beta.4
├─┬ @stryker-mutator/typescript-checker@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ ├─┬ @stryker-mutator/util@4.0.0-beta.4
├─┬ @stryker-mutator/webpack-transpiler@3.3.1
│ ├── @stryker-mutator/api@3.3.1 deduped
├── @types/mocha@5.2.7

Additional context

I haven't added any config for the typescript checker as the build command makes use of multiple tsconfig files for different parts of the code.

@gramster gramster added the ⁉ Question Further information is requested label Sep 9, 2020
@bartekleon
Copy link
Member

bartekleon commented Sep 9, 2020

check for sure if you have good packages, i can see some 3.3.1 versions here. + i think we are not using transpilers now (i dunno if it beta.4 tho). Also please provide full stryker.log file :)

and if possible it is helpful when you have some reproductive repo (unless its private or something) so its easier for us to debug :)

@gramster
Copy link
Author

gramster commented Sep 9, 2020

I pointed you to the repo I am running against, which together with the config file is everything you would need to repro (except for bug 2465, which I had to work around - the modified repo is at this fork/branch: https://github.com/gramster/vscode-python-me/tree/stryker). I can see that the Stryker process is still consuming CPU but I don't know what it is doing. Full log file is too large to attach but I don't think it will tell you anything more useful than what I posted above. Some of the old packages are there because I started with Stryker 3 and then switched to beta; I don't think they should have any impact but I can remove them.

@gramster
Copy link
Author

gramster commented Sep 9, 2020

I removed all the Stryker packages and then ran:

npm install --save-dev @stryker-mutator/core@next @stryker-mutator/mocha-framework@next @stryker-mutator/mocha-runner@next @stryker-mutator/typescript-checker@nex

which resulted in:

➜  vscode-python-me git:(stryker) ✗ npm ls | grep stryker
├─┬ UNMET PEER DEPENDENCY @stryker-mutator/core@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ ├─┬ @stryker-mutator/instrumenter@4.0.0-beta.4
│ │ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ │ ├── @stryker-mutator/util@4.0.0-beta.4 deduped
│ ├─┬ @stryker-mutator/util@4.0.0-beta.4
├── @stryker-mutator/javascript-mutator@4.0.0-beta.4
├─┬ @stryker-mutator/mocha-framework@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ └── @stryker-mutator/util@4.0.0-beta.4 deduped
├─┬ @stryker-mutator/mocha-runner@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ ├── @stryker-mutator/util@4.0.0-beta.4 deduped
├── @stryker-mutator/typescript@4.0.0-beta.4
├─┬ @stryker-mutator/typescript-checker@4.0.0-beta.4
│ ├─┬ @stryker-mutator/api@4.0.0-beta.4
│ ├── @stryker-mutator/util@4.0.0-beta.4 deduped
├─┬ @stryker-mutator/webpack-transpiler@3.3.1
│ ├─┬ @stryker-mutator/api@3.3.1

and stryker still gets stuck.

@gramster
Copy link
Author

gramster commented Sep 9, 2020

I broke in with a debugger and got this call stack:

image

It turned out it was stuck trying to process files created by Stryker itself in the reports/ directory. After deleting that folder it started making progress again. Stryker should probably be ignoring that directory.

It failed shortly after:

11:28:25 (59053) INFO Instrumenter Instrumented 768 source file(s) with 48482 mutant(s)
11:28:43 (59053) DEBUG TSConfigPreprocessor Rewriting file File {
  name: '/Users/grwheele/repos/vscode-python-me/tsconfig.json',
  _content: <Buffer 7b 0a 20 20 20 20 22 63 6f 6d 70 69 6c 65 72 4f 70 74 69 6f 6e 73 22 3a 20 7b 0a 20 20 20 20 20 20 20 20 22 62 61 73 65 55 72 6c 22 3a 20 22 2e 22 2c ... 957 more bytes>,
  _textContent: '{\n' +
    '    "compilerOptions": {\n' +
    '        "baseUrl": ".",\n' +
    '        "paths": {\n' +
    '            "*": ["types/*"]\n' +
    '        },\n' +
    '        "module": "commonjs",\n' +
    '        "target": "es2018",\n' +
    '        "outDir": "out",\n' +
    '        "lib": ["es6", "es2018", "dom", "ES2019"],\n' +
    '        "jsx": "react",\n' +
    '        "sourceMap": true,\n' +
    '        "rootDir": "src",\n' +
    '        "experimentalDecorators": true,\n' +
    '        "allowSyntheticDefaultImports": true,\n' +
    '        "strict": true,\n' +
    '        "noImplicitAny": true,\n' +
    '        "noImplicitThis": true,\n' +
    '        "noUnusedLocals": true,\n' +
    '        "noUnusedParameters": true,\n' +
    '        "noFallthroughCasesInSwitch": true\n' +
    '    },\n' +
    '    "exclude": [\n' +
    '        "node_modules",\n' +
    '        ".vscode-test",\n' +
    '        ".vscode test",\n' +
    '        "src/server/node_modules",\n' +
    '        "src/client/node_modules",\n' +
    '        "src/server/src/typings",\n' +
    '        "src/client/src/typings",\n' +
    '        "src/ipywidgets",\n' +
    '        "src/smoke",\n' +
    '        "src/test/datascience/extensionapi",\n' +
    '        "build",\n' +
    '        "out",\n' +
    '        "ipywidgets",\n' +
    '        "tmp"\n' +
    '    ]\n' +
    '}\n'
}
11:28:43 (59053) INFO ConcurrencyTokenProvider Creating 7 test runner process(es).
11:28:43 (59053) DEBUG Sandbox Creating a sandbox for files in /Users/grwheele/repos/vscode-python-me/.stryker-tmp/sandbox1581964
11:28:44 (59053) INFO Sandbox Running build command "npx gulp prePublishNonBundle" in the sandbox at "/Users/grwheele/repos/vscode-python-me/.stryker-tmp/sandbox1581964".
11:28:44 (59053) DEBUG Sandbox (using PATH: /Users/grwheele/repos/vscode-python-me/node_modules/.bin:/Users/grwheele/repos/node_modules/.bin:/Users/grwheele/node_modules/.bin:/Users/node_modules/.bin:/node_modules/.bin:/usr/local/bin:/Users/grwheele/.pyenv/shims:/Users/grwheele/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/opt/X11/bin:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Xamarin Workbooks.app/Contents/SharedSupport/path-bin)
11:30:06 (59053) ERROR Stryker an error occurred Error: Command failed with exit code 1: npx gulp prePublishNonBundle
[11:30:06] 'compile' errored after 1.32 min
[11:30:06] Error: TypeScript compilation errors
    at Pumpify.<anonymous> (/Users/grwheele/repos/vscode-python-me/.stryker-tmp/sandbox1581964/gulpfile.js:63:44)
    at Pumpify.emit (events.js:327:22)
    at Pumpify.EventEmitter.emit (domain.js:505:15)
    at finishMaybe (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:630:14)
    at afterWrite (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:492:3)
    at onwrite (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:483:7)
    at Pumpify.WritableState.onwrite (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:180:5)
    at Object.onceWrapper (events.js:421:28)
    at Pumpify.emit (events.js:315:20)
    at Pumpify.EventEmitter.emit (domain.js:505:15)
[11:30:06] 'prePublishNonBundle' errored after 1.32 min
[11:28:47] Using gulpfile ~/repos/vscode-python-me/.stryker-tmp/sandbox1581964/gulpfile.js
[11:28:47] Starting 'prePublishNonBundle'...
[11:28:47] Starting 'compile'...
src/client/activation/extensionSurvey.ts(275,2): error TS1160: Unterminated template literal.
src/client/activation/languageServer/analysisOptions.ts(265,233): error TS1002: Unterminated string literal.
src/client/common/installer/channelManager.ts(300,2): error TS1160: Unterminated template literal.
src/client/datascience/commands/commandRegistry.ts(1506,130): error TS1005: ')' expected.
src/client/datascience/commands/commandRegistry.ts(1506,135): error TS1005: ',' expected.
src/client/datascience/commands/commandRegistry.ts(1507,9): error TS1135: Argument expression expected.
src/client/datascience/editor-integration/cellhashprovider.ts(1170,2): error TS1160: Unterminated template literal.
src/client/datascience/interactive-window/identity.ts(110,118): error TS1005: ')' expected.
src/client/datascience/interactive-window/identity.ts(110,125): error TS1005: ',' expected.
src/client/datascience/interactive-window/identity.ts(111,7): error TS1135: Argument expression expected.
src/client/datascience/interactive-window/identity.ts(142,2): error TS1160: Unterminated template literal.
src/client/datascience/jupyter/jupyterUtils.ts(150,145): error TS1005: ')' expected.
src/client/datascience/jupyter/jupyterUtils.ts(150,146): error TS1005: ',' expected.
src/client/datascience/jupyter/jupyterUtils.ts(150,150): error TS1005: ',' expected.
src/client/datascience/jupyter/jupyterUtils.ts(150,163): error TS1005: ',' expected.
src/client/datascience/jupyter/jupyterUtils.ts(150,167): error TS1005: ',' expected.
src/client/datascience/jupyter/jupyterUtils.ts(202,2): error TS1160: Unterminated template literal.
src/client/testing/common/argumentsHelper.ts(378,132): error TS1005: ')' expected.
src/client/testing/common/argumentsHelper.ts(378,140): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,148): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,153): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,160): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,167): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,172): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,177): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,184): error TS1005: ',' expected.
src/client/testing/common/argumentsHelper.ts(378,188): error TS1005: '(' expected.
src/client/testing/common/argumentsHelper.ts(378,194): error TS1005: ';' expected.
src/client/testing/common/argumentsHelper.ts(378,211): error TS1005: ';' expected.
src/client/testing/common/argumentsHelper.ts(378,218): error TS1005: ')' expected.
src/client/testing/common/argumentsHelper.ts(378,221): error TS1005: ';' expected.
TypeScript: 31 syntax errors
TypeScript: emit succeeded (with errors)
    at makeError (/Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/node_modules/execa/lib/error.js:59:11)
    at handlePromise (/Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/node_modules/execa/index.js:114:26)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Sandbox.runBuildCommand (/Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/src/sandbox/sandbox.js:45:13)
    at async Sandbox.initialize (/Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/src/sandbox/sandbox.js:23:9)
    at async /Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/src/sandbox/sandbox.js:82:5
    at async MutantInstrumenterExecutor.execute (/Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/src/process/2-MutantInstrumenterExecutor.js:35:25)
    at async Stryker.runMutationTest (/Users/grwheele/repos/vscode-python-me/node_modules/@stryker-mutator/core/src/Stryker.js:32:44) {
  shortMessage: 'Command failed with exit code 1: npx gulp prePublishNonBundle',
  command: 'npx gulp prePublishNonBundle',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: '[11:28:47] Using gulpfile ~/repos/vscode-python-me/.stryker-tmp/sandbox1581964/gulpfile.js\n' +
    "[11:28:47] Starting 'prePublishNonBundle'...\n" +
    "[11:28:47] Starting 'compile'...\n" +
    '\u001b[31msrc/client/activation/extensionSurvey.ts(275,2): \u001b[39merror TS1160: Unterminated template literal.\n' +
    '\u001b[31msrc/client/activation/languageServer/analysisOptions.ts(265,233): \u001b[39merror TS1002: Unterminated string literal.\n' +
    '\u001b[31msrc/client/common/installer/channelManager.ts(300,2): \u001b[39merror TS1160: Unterminated template literal.\n' +
    "\u001b[31msrc/client/datascience/commands/commandRegistry.ts(1506,130): \u001b[39merror TS1005: ')' expected.\n" +
    "\u001b[31msrc/client/datascience/commands/commandRegistry.ts(1506,135): \u001b[39merror TS1005: ',' expected.\n" +
    '\u001b[31msrc/client/datascience/commands/commandRegistry.ts(1507,9): \u001b[39merror TS1135: Argument expression expected.\n' +
    '\u001b[31msrc/client/datascience/editor-integration/cellhashprovider.ts(1170,2): \u001b[39merror TS1160: Unterminated template literal.\n' +
    "\u001b[31msrc/client/datascience/interactive-window/identity.ts(110,118): \u001b[39merror TS1005: ')' expected.\n" +
    "\u001b[31msrc/client/datascience/interactive-window/identity.ts(110,125): \u001b[39merror TS1005: ',' expected.\n" +
    '\u001b[31msrc/client/datascience/interactive-window/identity.ts(111,7): \u001b[39merror TS1135: Argument expression expected.\n' +
    '\u001b[31msrc/client/datascience/interactive-window/identity.ts(142,2): \u001b[39merror TS1160: Unterminated template literal.\n' +
    "\u001b[31msrc/client/datascience/jupyter/jupyterUtils.ts(150,145): \u001b[39merror TS1005: ')' expected.\n" +
    "\u001b[31msrc/client/datascience/jupyter/jupyterUtils.ts(150,146): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/datascience/jupyter/jupyterUtils.ts(150,150): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/datascience/jupyter/jupyterUtils.ts(150,163): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/datascience/jupyter/jupyterUtils.ts(150,167): \u001b[39merror TS1005: ',' expected.\n" +
    '\u001b[31msrc/client/datascience/jupyter/jupyterUtils.ts(202,2): \u001b[39merror TS1160: Unterminated template literal.\n' +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,132): \u001b[39merror TS1005: ')' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,140): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,148): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,153): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,160): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,167): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,172): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,177): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,184): \u001b[39merror TS1005: ',' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,188): \u001b[39merror TS1005: '(' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,194): \u001b[39merror TS1005: ';' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,211): \u001b[39merror TS1005: ';' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,218): \u001b[39merror TS1005: ')' expected.\n" +
    "\u001b[31msrc/client/testing/common/argumentsHelper.ts(378,221): \u001b[39merror TS1005: ';' expected.\n" +
    'TypeScript: \u001b[35m31\u001b[39m syntax errors\n' +
    'TypeScript: emit \u001b[36msucceeded\u001b[39m (with errors)',
  stderr: "[11:30:06] 'compile' errored after 1.32 min\n" +
    '[11:30:06] Error: TypeScript compilation errors\n' +
    '    at Pumpify.<anonymous> (/Users/grwheele/repos/vscode-python-me/.stryker-tmp/sandbox1581964/gulpfile.js:63:44)\n' +
    '    at Pumpify.emit (events.js:327:22)\n' +
    '    at Pumpify.EventEmitter.emit (domain.js:505:15)\n' +
    '    at finishMaybe (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:630:14)\n' +
    '    at afterWrite (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:492:3)\n' +
    '    at onwrite (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:483:7)\n' +
    '    at Pumpify.WritableState.onwrite (/Users/grwheele/repos/vscode-python-me/node_modules/duplexify/node_modules/readable-stream/lib/_stream_writable.js:180:5)\n' +
    '    at Object.onceWrapper (events.js:421:28)\n' +
    '    at Pumpify.emit (events.js:315:20)\n' +
    '    at Pumpify.EventEmitter.emit (domain.js:505:15)\n' +
    "[11:30:06] 'prePublishNonBundle' errored after 1.32 min",
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}
11:30:06 (59053) DEBUG TemporaryDirectory Deleting stryker temp directory /Users/grwheele/repos/vscode-python-me/.stryker-tmp

@gramster gramster changed the title Execution doesn't progress after mutation Beta: Mutations are creating invalid code Sep 9, 2020
@gramster gramster changed the title Beta: Mutations are creating invalid code Beta: Stryker can get confused by its own reports directory Sep 9, 2020
@nicojs nicojs added 🐛 Bug Something isn't working and removed ⁉ Question Further information is requested labels Sep 9, 2020
@nicojs nicojs added this to the 4.0 milestone Sep 9, 2020
@nicojs
Copy link
Member

nicojs commented Sep 9, 2020

Awesome work you're doing here @gramster This is helping us a lot 🎉

It turned out it was stuck trying to process files created by Stryker itself in the reports/ directory. After deleting that folder it started making progress again. Stryker should probably be ignoring that directory.

I think this is a bit "by design" 😅 . We're currently using your own .gitignore file to ignore the files you're ignoring in your project. So we assume you're putting reports in your ignore file. Maybe we should add it during the stryker init command.

As for the strip comments preprocessor. It's actually a bug that is already fixed, but we didn't release a new version of the beta. Let me do that right now, curious to see if that fixes a lot of issues you're having.

EDIT: Just did a new release: v4.0.0-beta.5. It should fix this issue.

What we're basically doing, is create TS compile errors. No way around that, since we're instrumenting your TS code with mutants.

To disable type checks, we're adding // @ts-nocheck on top of the file. This allows typescript to transpile without errors. By default, this will happen for files matching this pattern: "{src,lib,test}/**/*.{ts,js,tsx,jsx}", so you probably also want change that to accomondate for your files:

For example:

{
  "disableTypeChecks": "**/*.ts"
}

See #2438 and the PR #2446 for more details.

@nicojs
Copy link
Member

nicojs commented Sep 17, 2020

@gramster I think this can be closed, or is there still something to be done here?

@nicojs
Copy link
Member

nicojs commented Sep 22, 2020

Closing this for now. Don't hesitate to contact me if you think it is a mistake 👍

@nicojs nicojs closed this as completed Sep 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants