When running with --retry, exit code should be 0 if a failed scenario passes on retry

[rishi-freshbooks]

Summary

When running cucumber with the --retry flag, the process will exit with a non-zero code if the scenario fails on first try but passes on a retry.

Expected Behavior

Running cucumber with --retry flag should exit with code 0 if scenarios pass on first try or within the specified number of retries .

Context & Motivation

The non-zero exit code causes our build job in JenkinsCI to fail when it should report as success.

[rishi-freshbooks]

@mattwynne does this capture the issue appropriately? Is this something that can be addressed in v3.0.0.pre.1?

[mattwynne]

That's great thanks @rishi-freshbooks. We'll do our best to fix this in a future pre-release of v3, yes.

[streetlogics]

@mattwynne any progress on this? Just encountering this issue myself - I love that I finally got the --retry flag working in circleci (note: I had to put the command at the end of the line otherwise a --format option that had an --out param seemed to swallow the retry attempt), however, once the retry started "working", circle still saw the run as a failure. 😞

[streetlogics]

@mattwynne - I took a really ugly pass at this (master...streetlogics:issue/retry-success-exit-code-0-1044) just to get something working. Definitely open to any thoughts you might have on improving this for the actual implementation

[mattwynne]

Awesome, thanks @streetlogics one of us will take a look ASAP

[garie]

Any update? My team is also interested in this fix as a non-zero exit code causes our Bamboo build to fail.

[garside]

Just ran into the --retry flag and the non-zero exit code for circleci. Just hoping to get a pulse check on the status of this retry-success feature that @streetlogics proofed.

@mattwynne, Is there an ETA for it and/or is there anything we could do to help push this across the finish line?

[brasmusson]

Yes, in non-strict mode the exit code should be 0 is all scenarios passes on retry, but in strict mode (option --strict) the exit code should not be 0 is any scenarios were retried.

I think a that first the summary should be updated with a new flaky category (as in 6 Scenarios (1 Passed, 1 Failed, 1 Undefined, 1 Pending, 1 Skipped, 1 Flaky)), then it is straight forward to implement the exit code logic. In non-strict mode the exit code should be 0 if and only if there no failed scenarios. In strict mode the exit code should be 0 if and only if there are no failed, flaky, pending or undefined scenarios.

@streetlogics patch does not handle the strict mode correctly, and as I just stated, I think that the exit code should be based on the summary (if the summary reports a failed scenario - then the exit code should be non-zero)