Functional test failure crashes intern when running through Grunt

[bmnielsen]

I have set up a couple of demo tests that just open Google's homepage and assert the page title. They're running as two separate functional test suites against a local Selenium server. With the correct expected value in the asserts, everything is fine and it reports the console output correctly.

However, if I modify one of the expected values in the asserts to simulate a test failure, Intern crashes out and stops reporting anything when it hits the failure. The test error bubbles up to Grunt, which outputs it as a warning.

If I run Intern from the command-line directly, I get the expected result.

The only options I'm specifying in the grunt config are runType: "runner" and the path to my config file, which is exactly what I'm specifying on the command-line.

[csnover]

Could you please provide an expected result & actual result example?

[bmnielsen]

Sure, here's a minimal set of files to reproduce. Also reproduces issue 46.

Gruntfile.js:

module.exports = function (grunt) {
    grunt.initConfig({
        intern: {
            all: {
                options: {
                    runType: 'runner',
                    config: 'tests/intern',
                    reporters: ['console']
                }
            }
        }
    });
    grunt.loadNpmTasks('intern');
};

tests/intern.js:

define({
    proxyPort: 9000,
    proxyUrl: 'http://localhost:9000/',
    capabilities: {
        'selenium-version': '2.33.0'
    },
    environments: [
        { browserName: 'chrome' }
    ],
    maxConcurrency: 3,
    useSauceConnect: false,
    webdriver: {
        host: 'localhost',
        port: 4444
    },
    loader: {
    },
    suites: [],
    functionalSuites: [ 'tests/test' ],
    excludeInstrumentation: /^tests\//
});

tests/test.js:

define([
    'intern!object',
    'intern/chai!assert'
], function (registerSuite, assert) {
    registerSuite({
        name: 'google homepage',
        'fails': function () {
            return this.remote.get('http://www.google.com')
                .title()
                .then(function (title) {
                    assert.strictEqual(title, 'Goggle');
                });
        },
        'succeeds': function () {
            return this.remote.get('http://www.google.com')
                .title()
                .then(function (title) {
                    assert.strictEqual(title, 'Google');
                });
        }
    });
});

Expected result is one passed test and one failed test reported to console.

Result when running node node_modules/intern/runner.js config=tests/intern reporters=console:

Listening on 0.0.0.0:9000
0/0 tests passed
FAIL: main - google homepage - fails (1596ms)
Error: expected 'Google' to equal 'Goggle'
expected 'Google' to equal 'Goggle'
FAIL: main - google homepage - succeeds (2ms)
Error: expected 'Google' to equal 'Goggle'
expected 'Google' to equal 'Goggle'
0/2 tests passed
0/2 tests passed

So I get a report to the console, but it's reporting both tests failed (issue 46).

Result when running grunt intern:

Running "intern:all" (intern) task
Listening on 0.0.0.0:9000
>> 0/0 tests passed
Warning: FAIL: main - google homepage - fails (6948ms) Use --force to continue.

Aborted due to warnings.

Once intern hits the test failure, it crashes out and doesn't report anything further, and Grunt aborts.

Another note is that in both cases, the browser window is left open.

If I fix the failing test, both commands give the expected result and the browser window is closed.

An additional thing I noticed while setting this up: with the "runner" reporter, running via grunt doesn't output anything even if the tests succeed.

[csnover]

This is the same issue as #46. A pull request exists at #69. It will be tracked there. Thanks for the report!

[bmnielsen]

Please re-read, this isn't the same issue as #46. The issue I'm reporting here is that running intern through Grunt gives a different result from running it directly via the Node command line, namely that intern crashes and doesn't report anything.

Issue #46 is only tangentially related to this, since the result from the Node command line is also incorrect. But the point here is that running it through Grunt doesn't work.

[csnover]

Grunt is supposed to stop proceeding when a test fails; I will be sure to check it once #46 is fixed to make sure it is working properly. The manner in which the failure occurs here is indicative of the problems caused by #46. If I am a dolt and it turns out to not be the root cause, I will reopen this.

[csnover]

Having fixed #46 and coming back to verify this, I am actually not able to reproduce your reported output.

Intern 1.1.0:

$ grunt intern
Running "intern:all" (intern) task
Listening on 0.0.0.0:9000
>> 0/0 tests passed
Warning: FAIL: main - google homepage - fails (2409ms)
Error: expected 'Google' to equal 'Goggle'
expected 'Google' to equal 'Goggle'
FAIL: main - google homepage - succeeds (3ms)
Error: expected 'Google' to equal 'Goggle'
expected 'Google' to equal 'Goggle'
0/2 tests passed
0/2 tests passed Use --force to continue.

Aborted due to warnings.

So, wrong, but not what is reported in this ticket.

On the current master branch things look even better:

$ grunt intern
Running "intern:all" (intern) task
Listening on 0.0.0.0:9000
>> 0/0 tests passed
>> PASS: main - google homepage - succeeds (2185ms)
Warning: FAIL: main - google homepage - fails (1640ms)
Error: expected 'Google' to equal 'Goggle'
expected 'Google' to equal 'Goggle'
1/2 tests passed
1/2 tests passed Use --force to continue.

Aborted due to warnings.

Could you test again using the master branch and let me know what your results are? There is another change in master that I am wondering fixes the issue (Intern 1.1 and earlier quit the node process immediately once the tests are done running; 1.2 and later allow it to exit gracefully, so if there was stuff left in a buffer that had not been written out in 1.1, it may be written out correctly in 1.2). I’ll reopen this for now so it is still on radar until you’ve had a chance to test.

If you test on master and it still continues to occur, could you please verify your Grunt version and operating system? Thanks!

[bmnielsen]

Looks like master works now. Thanks!