--spec order is not respected. Defaults to alphabetical execution.

[diggabyte]

Ordering of the --spec argument is not respected. This is critical for our use case as it pertains to optimization / balancing of our suites to distribute evenly across workers. This should be a simple fix... just don't sort the spec arguments by alpha.

Current behavior

Does not respect the order of --spec, always runs specs in alphabetized order

 $ yarn cypress run --spec \
   cypress/integration/defect/c.ts,cypress/integration/defect/b.ts,cypress/integration/defect/a.ts

Input specs are a specific order:

Actual spec execution is alphabetized:

`

Desired behavior

Respect the order passed via --spec and do not alphabetize the run order

Test code to reproduce

Just rename some tests, easy to reproduce

Versions

6.5.x, 6.6.x, possibly much, much older

[diggabyte]

Verification the actual run order is alphabetized.. so it's not just presentation of the test results:

[jennifer-shehane]

This is by design. Duplicate of #390

[diggabyte]

@jennifer-shehane we should be able to place failing tests first to save developers time. CI can be painfully slow waiting for results if you have a failing test that happens to be alphabetically last. Additional context provided on the original issue here: #390 (comment)

[jennifer-shehane]

@diggabyte As part of Cypress’s new pricing, we’ve included the ability to cancel test runs when a test fails. This setting is accessible from the Dashboard for organizations starting at the Business Plan.

This offers a solution for those running tests using the Cypress Dashboard - and also ensures a parallelized run is cancelled so that all parallel running specs will also be cancelled to save time on the run.

To get this feature, you will need to update to Cypress 6.8.0 and also be a member of an organization subscribed to a Business Plan.

---

This feature was implemented with parallelized runs in the Dashboard in mind since this was the hardest use case to address. We had to build this feature specifically to continue to receive all of the tests in a cancelled run to ensure proper reporting. Now that we have a mechanism to cancel runs across these channels of communication, we can now consider a way to initiate cancelling test runs when a test fails from the Test Runner when not recording to the Dashboard. (Likely this would be implemented by some CLI flag or config specifying `cancelOnFailures: true)

See this issue #518 on addressing the use case of cancelling test runs when a test fails from the Test Runner when not recording to the Dashboard.