[Bug]: reporter onRunComplete not triggered when jest runner crashed in watch mode

[connectdotz]

Version

28.1.3

Steps to reproduce

1. clone https://github.com/marr/jest-async-fail
   2. run yarn install
   3. create a simple reporter with onRunComplete and onRunStart methods if you are not using vscode-jest extension in vscode
   4. run jest command: with the custom reporter. For example, with the vscode-jest extension:
   yarn test --watchAll --no-coverage --reporters default --reporters "/your-vscode-extension-path/vscode-jest/out/reporter.js"

you should see something like this:

  ● Test suite failed to run

    Cannot find module 'vue-template-compiler'
    Require stack:
    - /jest-async-fail-main/node_modules/@vue/vue2-jest/lib/process.js
    - /jest-async-fail-main/node_modules/@vue/vue2-jest/lib/index.js
    - /jest-async-fail-main/node_modules/jest-util/build/requireOrImportModule.js
    - /jest-async-fail-main/node_modules/jest-util/build/index.js
    - /jest-async-fail-main/node_modules/jest-config/build/getCacheDirectory.js
    - /jest-async-fail-main/node_modules/jest-config/build/Defaults.js
    - /jest-async-fail-main/node_modules/jest-config/build/normalize.js
    - /jest-async-fail-main/node_modules/jest-config/build/index.js
    - /jest-async-fail-main/node_modules/jest-cli/build/init/index.js
    - /jest-async-fail-main/node_modules/jest-cli/build/cli/index.js
    - /jest-async-fail-main/node_modules/jest-cli/build/index.js
    - /jest-async-fail-main/node_modules/jest-cli/bin/jest.js
    - /jest-async-fail-main/node_modules/jest/bin/jest.js

      at Object.<anonymous> (node_modules/@vue/vue2-jest/lib/process.js:1:29)

onRunStart


Test Suites: 0 of 2 total
Tests:       0 total
Snapshots:   0 total

The onRunComplete, which should output "onRunComplete", is never invoked.

Expected behavior

the reporter should receive onRunComplete callback with at least testResult.testExecError populated.

Actual behavior

reporter onRunComplete was never called.

Additional context

We discovered this issue during the investigation of jest-community/vscode-jest#894. The culprit seems to be in TestScheduler where it catches runner exception and rethrow if it's in watch mode and not interrupted.

It didn't seem right as this resulted in reporters not knowing what is happening: it received the onRunStart but nothing else afterward. It is also inconsistent that other watch mode runs that are always reported with (onRunStart, onRunComplete) pair. It seems we could capture this error in TestResult.testExecError and report it via onRunComplete.

I could write a PR if you agree with the fix.

Environment

System:
    OS: macOS 12.5.1
    CPU: (12) x64 Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
  Binaries:
    Node: 16.13.2 - ~/.nvm/versions/node/v16.13.2/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 8.1.2 - ~/.nvm/versions/node/v16.13.2/bin/npm
  npmPackages:
    jest: ^28.1.3 => 28.1.3

[connectdotz]

After further investigation, the crash occurred even earlier than I originally thought: in createScriptTransformer, right after we dispatched onRunStart...

Basically, there is a situation when runner failed due to any runtime/execError within or prior, it will fail to notify reporters. Does anybody know if that was the intended behavior? If not, I think we can capture those exceptions and report back with the onRunComplete, plus an additional attribute like AggregatedResult.execError, similar to the TestResult.testExecError. Or to be safe and abundantly clear, we can even create a new API Reporter.onRunFail if desired...

Does that make sense? Is there any reason we don't want to do this?

[connectdotz]

This issue has prevented us to know that jest run has failed and no results will ever be produced. Since in watch mode, the process will not exit in this situation, our result processing code and reporters will be waiting indefinitely.

@SimenB will you consider accepting a PR for this issue?

[SimenB]

Definitely!

[SimenB]

https://github.com/facebook/jest/releases/tag/v29.0.3

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.