Failing to load a (esm) module does not fail the test suite

[jehon]

When loading (esm) modules, if one module has an error in it (example: it could not import another module due to a typo in the import statement), the test should fail.

Karma config:

module.exports = function (config) {
    config.set({
        ...
        files: [
            { pattern: 'tests/*-test.mjs', **type: 'module'** },
            { pattern: '**/*', watched: false, included: false },
        ],
    });
};

Example code:

import { something } from './somewhere.js';
...

If "somewhere.js" does not exists, the test should fail.

Current Behavior

Today, the test pass, skipping the "failingtest.js" as it did not exists, and report the test suite as successful.

09 11 2020 11:52:49.258:INFO [karma-server]: Karma v5.2.3 server started at http://localhost:9876/
09 11 2020 11:52:49.259:INFO [launcher]: Launching browsers ChromiumHeadless with concurrency unlimited
09 11 2020 11:52:49.263:INFO [launcher]: Starting browser ChromiumHeadless
09 11 2020 11:52:49.542:INFO [Chrome Headless 86.0.4240.183 (Linux x86_64)]: Connected on socket pCX5O55W4Zk2P-JAAAAA with id 38747259
09 11 2020 11:52:49.592:WARN [web-server]: 404: /base/tests/something.js
Chrome Headless 86.0.4240.183 (Linux x86_64): Executed 1 of 1 SUCCESS (0.003 secs / 0.001 secs)
TOTAL: 1 SUCCESS

While obviously, the second test did not run and the whole run should be in error

I have made a small project to show the problem:
=> https://github.com/jehon/example-jasmine
npm run test (single pass)
(npm run test-debug to keep it open)

The output of the test is available here: https://travis-ci.com/github/jehon/example-jasmine/builds/200293863

My Environment:

• Karma: 5.2.3
• Karma-jasmine: 4.0.1

[jehon]

I find here an error handler:

https://github.com/karma-runner/karma/blob/master/static/karma.js#L185

  // error during js file loading (most likely syntax error)
  // we are not going to execute at all. `window.onerror` callback.
  this.error = function (messageOrEvent, source, lineno, colno, error) {

That is not called when a module script fail to load.

[jehon]

After digging around, I find this:

contextWindow.onerror = function () {
   return self.error.apply(self, arguments)
}

(

karma/static/context.js

Line 214 in 6cd5a3b

contextWindow.onerror = function () {

)
which is not triggered when a module fail to load.

One solution would be to change

scriptTags.push(`<script type="${scriptType}" src="${filePath}" ${crossOriginAttribute}></script>`)

into

scriptTags.push(`<script onerror="throw 'Error loading ${filePath}'" type="${scriptType}" src="${filePath}" ${crossOriginAttribute}></script>`)

https://github.com/karma-runner/karma/blob/master/lib/middleware/karma.js#L196

This lead to this error, which is acceptable to me:

Chrome Headless 86.0.4240.198 (Linux x86_64) ERROR
An error was thrown in afterAll
Uncaught Error loading tests/failing-test.mjs
http://localhost:9876/context.html:2463
Chrome Headless 86.0.4240.198 (Linux x86_64): Executed 1 of 1 ERROR (0.005 secs / 0.001 secs)

If you agree with this, I will make a pull request

[karmarunnerbot]

🎉 This issue has been resolved in version 6.0.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀