Option to abort on first failure

[KittyGiraudel]

Current behavior:

All tests are run even if one fails.

Expected behavior:

Providing an option to abort the test suite as soon as a test fails.

How to reproduce the current behavior:

Make a test fail.

Environment

• Cypress Version: 0.19.2

[01taylop]

The use-case I have for this is that I need my authentication tests to pass first - if they don't all subsequent tests will fail. Although the other tests will fail in the before hooks, it still slows things down unnecessarily (30 seconds per test file).

I have to name my test 1auth so that it runs first and whitelist a cookie which contains a token required for the other tests to pass.

Cypress.Cookies.defaults({
  whitelist: 'token',
})

I was hoping to do something like this in my auth tests to stop running tests on fail;

  before(() => {
    Cypress.on('fail', () => {
      Cypress.runner.stop()
    })
  })

It does stop the test runner, but it doesn't fail any tests so I get 3/3 tests passed and a green build which is not the intended outcome. I also tried adding am intentional failing test before stopping cypress, but then the test runner continues.

[danh91]

I have a temporary solution that I think can help someone.

I just add an after each in my utilities file imported by all my specs.

afterEach(function() {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

I take advantage of Mocha offering a way to check the state of the last test.
And the nice part is that the process exit with a failing code (not 0)

[sanderdejong88]

@danh91 I've been following this thread for months, and finally someone found a way! Awesome!
I've put your code in the support file which also seems to work.

[harvitronix]

This no longer works for me with Cypress 3.x.

[jennifer-shehane]

@harvitronix We changed how specs are run in 3.0 so that the files are run isolated. This means the before runs before each spec file - so I am guessing that it is stopping and failing that 1 spec file and then continuing to the next file. Is this what is happening?

[brian-mann]

Yes we've waited to implement this feature due to how aborting is much more complex now.

We likely need to allow for multiple abort strategies

abortStrategy: 'spec' | 'run'

One would abort just the spec, one would abort the entire run.

This becomes even more complex with parallelization.

[harvitronix]

@jennifer-shehane that's exactly right. Totally understand it gets much more complex now. Would love to have the options you describe, @brian-mann.

[mohsenny]

I tried your solutions and the runner correctly stops.
However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s.
Any suggestion?

An example:

As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

[drumheadrob]

Hi, loving Cypress, but not being able to stop our CI build for a failing test is stopping me from being able to use this in CI, and will especially be a problem in CD.

I've searched quite thoroughly and hope I'm missing something somewhere, but it seems like this is related to this thread?

If so, is there any progress / planned work on this solution? Cheers.

[bahmutov]

@drumheadrob your issue is very different - you are saying that the cypress run does NOT exit with code 1 after all tests complete and some fail?!

[jennifer-shehane]

We are currently working on the underlying processes needed to 'cancel' a run and any other queued spec files, which is a precursor to the work needed to 'abort' the rest of the tests on first failure.

[pacop]

Some news right here? I am facing the very same problem, we were not able to use cypress on our CI.

[nantoniazzi]

Are you still working on this feature ?
The more we add tests with Cypress, the more it becomes unusable in our CI

[cmcnicholas]

We have the same situation, paying for cypress dashboard on private repos but our tests (now ~400 or so) are taking more and more time to complete. Much needed to stop early so our pipeline is freed up for the next PR to execute.

[vladarefiev]

It will be cool feature

[Josh68]

Another hack for making the spec fail (as I'd expect):

if (
  this?.currentTest?.state === "failed" // plus more conditions for retries
) {
...
  if ("state" in cy && typeof cy.state === "function") {
    cy.state("test").state = "failed";
  }
...
}

[csvan]

Just a note: the above workaround implies you are not using arrow functions for your test definitions.

[Josh68]

Just a note: the above workaround implies you are not using arrow functions for your test definitions.

Correct, it should be clear to anyone following along with this that for Mocha.Context to be available in the callback, you can't use an arrow function. Also, if you want to write a function you can pass by name as a callback, type it like so:

const failEarly: Mocha.Func = function failEarly() { // could also be anonymous
  // `this` is Mocha.Context
  ...
}

[Josh68]

Another hack for making the spec fail (as I'd expect):

if (
  this?.currentTest?.state === "failed" // plus more conditions for retries
) {
...
  if ("state" in cy && typeof cy.state === "function") {
    cy.state("test").state = "failed";
  }
...
}

I'm now finding this really doesn't do anything, AFAICT, and it looks suspiciously wrong anyway. I'm going to stop jumping through hoops assuming there are things I can do that aren't in the public API.

[Josh68]

@jennifer-shehane, can you comment on whether any of these strategies ought to work with v12+? I've been upgrading from v9, and at this point, don't have a strategy that aborts early on a failed test, marks the spec as failed, marks remaining tests as either Pending or Skipped, and then moves on (some may want the entire suite to stop, but we weren't doing that in the past).

Thank you

[Josh68]

Based on this example by @drumslave-git, I came up with something that I think is working globally:

export const failEarly = (runModeOnly = true) => {
  if (runModeOnly && Cypress.config("isInteractive")) return; // headless runs only
  const failedTest: Record<string, string | undefined> = {};
  beforeEach(function() {
    const specName = Cypress.spec.name;
    if (failedTest[specName]) {
      cy.log(
        `*** Spec has failed. You will see an error emitted from beforeEach, but only need to fix failing tests ***`
      );
    }
    cy.wrap(failedTest[specName]).should("be.undefined");
  });
  afterEach(function() {
    const retryCount = Cypress.currentRetry;
    const specName = Cypress.spec.name;
    if (
      this?.currentTest?.state === "failed" &&
      Number.isInteger(retryCount) &&
      getRunTypeRetries() <= retryCount
    ) {
      failedTest[
        specName
      ] = `Previous test, "${this.currentTest.title}", failed. This spec will be aborted.`;
    }
  });
};

getRunTypeRetries looks like this

const getRunTypeRetries = () => {
  const retryConfig = Cypress.config("retries");
  let configuredRetries = 0;
  if (typeof retryConfig === "number" && Number.isInteger(retryConfig)) {
    configuredRetries = retryConfig;
  }
  if (typeof retryConfig === "object" && null !== retryConfig) {
    if (
      retryConfig.runMode &&
      Number.isInteger(retryConfig.runMode) &&
      !Cypress.config("isInteractive")
    ) {
      configuredRetries = retryConfig.runMode;
    }
    if (
      retryConfig.openMode &&
      Number.isInteger(retryConfig.openMode) &&
      Cypress.config("isInteractive")
    ) {
      configuredRetries = retryConfig.openMode;
    }
  }
  return configuredRetries;
};

I imported failEarly into my e2e.ts and called it with

const runModeOnly = !Cypress.env("BROWSER_FAIL_EARLY") == true;
failEarly(runModeOnly);

with the idea that I can add CYPRESS_BROWSER_FAIL_EARLY=true before a cypress:open, if desired.

Note, I'm using cypress-terminal-report to display those cy.logs on headless runs

[w5l]

Options with Cypress.runner.end() or Cypress.runner.stop() don't work for me as they stop cypress before uploading test results. After some puzzling I came up with the following. This requires no extra setup, magic variables etc, just a simple afterEach call.

If the current test is failed, it cancels all subsequent tests within the same container. Since I only use one level of nesting (ie describe( it(), it(), it(), ...)), this is enough. All tests (within the spec) after a failed are now skipped.

This uses the Mocha bail option.

Enables or disables bailing on the first failure.

Works either on the global Cypress e2e.ts (or .js) level, or in individual specs:

before(() => {
  // Skip subsequent tests in spec when one fails.
  (cy.state('runnable').ctx as Mocha.Context).currentTest.parent.bail(true);
});

---

Update: Note that this is a fix for an older Cypress version version, recent comments suggest it no longer works, see #518 (comment)

Update 2: I was told that cy.runnableCtx() can be used instead of the deprecated cy.state(), resulting in the command cy.runnableCtx().currentTest.parent.bail(true);. That function is defined here:

cypress/packages/driver/src/cypress/cy.ts

Lines 1075 to 1079 in 22aacb1

	private runnableCtx (name) {
	ensureRunnable(this, name)
	return this.state('runnable').ctx
	}

Note that it is marked private so it can be changed at any time. Haven't tested myself, ymmv. Can't be much worse than using DOM selectors to find and click a UI button to stop, really 😉

[scmrus]

Is that the solution for 12.6.0 and later: https://docs.cypress.io/guides/guides/command-line#auto-cancel-after-runs ?

[Josh68]

Is that the solution for 12.6.0 and later: https://docs.cypress.io/guides/guides/command-line#auto-cancel-after-runs ?

Thanks for pointing this out. Will give it a try when we have a chance to upgrade

[BloodyRain2k]

I've spent half the day trying to find a way to prevent Cypress from continuing after failure, simply to preserve the debug data that what I'm testing left me for debugging.

After what feels like a dozen different approaches that didn't work at all or not for my case, I went and bruteforced it this way.

    if (Cypress.env("pause_after_fail")) {
      Cypress.on("test:after:run", (attrs, test) => {
        if (test.isFailed()) {
          const stop = window.top?.document.querySelector(".reporter button.stop") as HTMLElement;
          stop.click();
        }
      });
    }

What I did NOT expect was this to work.

"Best case" expectation was that it does click the "Stop" button, but keeps "rolling" from momentum (it's dozen Promise-chains).
But it doesn't. It just stops instantly and I can inspect my debug output without any worry.

If you're not using TypeScript, you can even reduce it to just .querySelector()?.click().
That storing as HTMLElement was simply so TypeScript would shut up about .click() not existing.

[machineghost]

before(() => {
  // Skip subsequent tests in spec when one fails.
  (cy.state('runnable').ctx as Mocha.Context).currentTest.parent.bail(true);
});

It seems cy.state no longer exists, so this solution is no longer viable.

Is that the solution for 12.6.0 and later: https://docs.cypress.io/guides/guides/command-line#auto-cancel-after-runs ?

It seems that auto-cancel-after-runs only works with the Cypress Cloud, and if you try to just cypress run --auto-cancel-after-failures 1 you get:

You passed the --ci-build-id, --group, --tag, --parallel, or --auto-cancel-after-failures flag without also passing the --record flag.

The --auto-cancel-after-failures flag you passed was: 1

These flags can only be used when recording to Cypress Cloud.

But the good news is that BloodyRain2k's solution did work. You can either gate it with an environment variable (as in the original code), or make a function out of it:

export default function enableFastFail() {
  Cypress.on('test:after:run', (attrs, test) => {
    if (test.isFailed()) {
      const stop = window.top?.document.querySelector(
        '.reporter button.stop'
      ) as HTMLElement;
      stop.click();
    }
  });
};

That way, when you want the runner to stop after the first fail, you just temporarily import enableFastFail and run it in the root of your test file. Thanks BloodyRain2k!

[NigelCunningham]

This works for me with Cypress 13.3.2

context('Context Name', () => {
  describe('Tests', () => {
    afterEach(() => {
      if (cy.runnableCtx().currentTest.isFailed()) {
        const stop = window.top?.document.querySelector(".reporter button.stop");
          stop.click();
      }
    })
  })
})

[vitaliy4us]

As far as I can see the most of solutions are about Cypress Runner:
const stop = window.top?.document.querySelector(".reporter button.stop")
But in this case it is almost useless. Headless mode is the point where we really need it, as it is used in CI tests runs and can save a lot of execution time. Especially for long e2e tests

[NigelCunningham]

Yes, it worked for me.

[philly-vanilly]

    if (test.isFailed()) {
      const stop = window.top?.document.querySelector(
        '.reporter button.stop'
      ) as HTMLElement;
      stop.click();
    }

I had a problem with this solution as it prevented tests from being retried. If anyone else also suffers from this, you can filter error messages on the test as a temporary solution (long term solution would be ditching Cypress):

afterEach(() => {
  const currentTest = ((cy as any).runnableCtx() as Mocha.Context).currentTest;

  // let login tests retry, do not abort
  if (currentTest.isFailed() && !currentTest.err?.message.includes('[data-cy="user-menu"]')) {
    const stop = window.top?.document.querySelector('.reporter button.stop');
    stop.click();
  }
});

[seyfer]

@mihhail-m I bet you meant playwright as an alternative

[philly-vanilly]

If someone doesn't want a feature, you can submit a million PRs, but it won't do anyone any good.

@machineghost

Actually, since Cypress is open source with MIT license, thus you are allowed to clone and manipulate the code, I would welcome if someone made changes to Cypress in form of a https://www.npmjs.com/package/patch-package file and posted it in this thread. pnpm users don't even need patch-package. Surely a feature that simple should not require too many files changed.

[EugeneSEP]

I'm not sure if this is the right place to ask, but I'm hoping someone here might have had a similar experience and could offer some advice. I have cypress, allure reporter and cypress-fail-fast
plugin
I'm encountering an issue with the Allure reporter in Cypress. I've set it up along with the fast-fail plugin, but my Allure report is only displaying failed and passed tests, excluding others.

How can I ensure that all tests, regardless of their outcome, are included in the report? Any guidance would be appreciated!