Continue on failure does not work if assert is wrapped in a method

[delmaamoun]

I have a step that calls a method to do the verification. When the assert fails in the method, it throws an exception and it's not caught by the continue on failure on the step level.

@step("Should Continue")
@continue_on_failure
def should_continue():
    failassert()

def failassert():
    assert False, "Failed!"

Steps to reproduce

1. create a step with "@continue_on_failure"
2. create a method which has "assert" and call that method within the step
3. run the scenario with the above step

Result: Scenario DOES NOT continue on failure
Expected: Assertion error should be handled.

Gauge version: 1.0.3
Commit Hash: ff6c0c3

Plugins
-------
html-report (4.0.6)
java (0.6.9)
python (0.3.5.nightly-2018-12-04)
screenshot (0.0.1)
spectacle (0.1.3)

[NivedhaSenthil]

Works as expected with given versions, remaining steps in scenario is executed even when exception happens in another function. Can you please share a project to replicate this ?

[delmaamoun]

My bad, it breaks the rest of the step, so if there is a table attached to it, it breaks that loop as well. Kind of inconvenient though.
I uploaded an example here: https://github.com/delmaamoun/gauge-continue

[NivedhaSenthil]

That is the expected behaviour of @continue_on_failure. If any exception occurs in a step marked as @continue_on_failure, it will continue to run other step and the remaining part from the point of exception in the current step will not be executed.

[delmaamoun]

I still see it as an impediment for data-driving. Why is the expected behavior to break the step especially in cases of assertion errors?

[sriv]

The continue_on_failure feature is meant to be used when a scenario needs to execute subsequent steps even if the current step fails. Please see this reference: https://docs.gauge.org/latest/writing-specifications.html#continue-on-failure

What you seek is perhaps a step implementation to continue even if there is a failure. I think you could benefit from a "soft assert", where your step implementation could accumulate all the failures and return back only after implementing it.

Please see how we do soft assert in our tests (the example is in java, but it should be easy enough to do something similar in python) - https://github.com/getgauge/gauge-tests/blob/master/src/test/java/com/thoughtworks/gauge/test/implementation/ProjectInit.java#L49-L60

[delmaamoun]

thanks @sriv, I used the delayed_assert package with expect().
It does the job.