Feature request: Fail-fast option

[sholladay]

Problem

For users of Continuous Integration / Delivery systems, the excess testing and build time caused by multiple test failures can be very undesirable, such as for a shared dev branch where speed is important and time-to-deploy is compounded and queued many times over. It can also consume precious resources from per-hour test clouds like BrowserStack and SauceLabs.

Solution

Provide an opt-in configuration option to halt at the first test failure. A relevant reference is the fail_fast option for RSpec or Mocha's --bail.

Suggested requirements

• When a test fails, all further tests and suites are aborted.
• Any tests not run due to a test failure are counted as skipped.
• When tests are skipped due to a test failure, a warning is logged to explain them.
• This option is off by default, to avoid confusion.

Stretch goal: make it configurable so that it can be told to only abort the rest of the current suite or only the rest of the unit tests, but not functional tests, etc.

[vladikoff]

Good suggestion, I do agree that this is gonna useful for CI, especially if you retry tests with something like travis_retry.

[sholladay]

It can be even better. If Intern natively supports retries, it doesn't have to repeat the entire lot of suites or even restart the process. In my ideal setup, CI would probably retry a test once or twice before considering it truly failed and once that had been decided, would not continue. So combining these is indeed powerful, especially if it's handled within Intern itself.

[vladikoff]

@csnover in #368 (comment) you mentioned that in Intern 3 it should be possible to pause flow, what this help this issue?

[csnover]

Yes it will help, it should be much easier to bail out of running any more tests now

[vladikoff]

any pointers on how to do that? I think @miketaylr wanted to hook that up
On Aug 11, 2015 3:17 PM, "Colin Snover" notifications@github.com wrote:

Yes it will help, it should be much easier to bail out of running any more
tests now

—
Reply to this email directly or view it on GitHub
#413 (comment).

[csnover]

In lib/Suite.js, if a test fails and fast fail is turned on then don’t run any more tests. This changed code also needs to account for sub-suites: a parent suite should not continue to run either if a child suite failed. That’s pretty much it.

[airtonix]

Any way I can enable fail fast but not have my terminal spammed with all the skipped tests?

NVM, i made my own reporter:

// ./test/e2e/support/FailFastReporter.js

/* global define */

define([
	'intern/lib/reporters/Runner',
], function (Runner) {

	function extend (target, source) {
		target.prototype = Object.create(source.prototype);
		target.prototype.constructor = target;
		return source.prototype;
	}

	var FailFastReporter = function (config) {
		Runner.call(this, config);
	};

	var parent = extend(FailFastReporter, Runner);

	FailFastReporter.prototype.testSkip = function () {};

	return FailFastReporter;

});

[jason0x43]

That's the spirit!

That's a good idea -- having an option to hide skipped tests would actually be pretty handy.