Accepting some test failures as success

[manu-st]

🚀 Feature Proposal

Add the ability to mark a test ok to fail:

test.failing (name, fn, timeout)
test.failing.each (table) (name, fn, timeout)
test.failing.only (name, fn, timeout)
test.failing.only.each (table) (name, fn, timeout)
test.failing.skip (name, fn, timeout)
test.failing.skip.each (name, fn, timeout)

Motivation

When you write tests, you want to run them all the time. With the current jest behavior, you get a successful run when all tests are passing which is great. However, sometime you may end up writing tests that will fail until a defect is fixed or a feature is implemented. In real life, you would ensure the test would pass, but in practice you want the test to be present but be marked as ok to fail until you actually fix/implement it.

Example

If a test is marked this way and it is failing, then jest will not return an error code. This is accepted.

If a test is marked this way and it is passing, then jest will return an error code as it is now passing and the failing option should be removed if this is expected (ideal situation) or the reason of the success should be investigated.

Pitch

Let's you write way more tests at the beginning without having them passing and being notified as soon as they are passing. At this point you should check why they are passing.

Overall, it makes jest a better regression testing tool as regression does not mean tests should always pass, but it means that failed tests should continue to fail unless fix/feature was implemented.

[jeysal]

Thanks for the suggestion! @SimenB @thymikee what do you think about this?
Personally, I'm not sure it adds that much value over the existing option of using skip. I also think it would be a bit hard to visualize in the reporter if pass becomes fail and fail becomes pass.

[manu-st]

Unlike skip, failing will always run the tests. Also skip doesn't provide a reason for the skip (it could be that it fails, or that it needs to be run manually, or ...) unless documented in the test itself.

Many times, I've worked on defects fixing a bug and then realizing that I fixed another one. If I had put the test in skip mode, I would not have known about it. And when this happened, I performed some analysis to ensure that the newly passing test is indeed passing because of the change I made and not a side effect of the change I made (meaning that I hid the original bug but it is still present).

[thymikee]

It can be implemented in userspace, so once you do it and prove it's useful for more developers out there, we can consider adding it to the core :).
Also please add this to awesome-jest repo once it's done!

Other than that you can still write expectations that are meant to fail.

[SimenB]

I think it's a great idea, and would love to see it in jest. See #4627

[thymikee]

Oh my, I somehow missed or didn't remember that 😅

[manu-st]

I was searching the issues for this but could not find it. Glad I'm not the only one who thought about this and interestingly enough we choose the same way to accomplish this. I'll monitor the other issue now.
Thanks!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.