[Feature]: Add expect.soft for setup and soft assertions

[marionebl]

🚀 Feature Proposal

A new method soft on the expect object giving test authors fine grained control over the control flow of their test case. An assertion decorated with .soft continues execution of a test case after assertion's failure to gather more data e.g. via remaining assertions in the case.

Motivation

This provides new tools to test authors striving to write more expressive and easier to debug tests.

Example

I often encounter test case with intermediary assertions designed to ensure a precondition relevant to the final assertion is met.

test('result is as expected given precondition is met', () => {
  const precondition = setup();
  expect(precondition).toBe(met);
 
  const result = fn(precondition);
  expect(result).toBe(asExpected)
})

This works but is sub-optimal in so far as in the case of a precondition failure useful debugging information is lost as execution stops after the first failed assertion:

  ● result is as expected given precondition is met

    expect(received).toBe(expected) // Object.is equality

    Expected: met
    Received: notMet

      20 |
      21 | test('an assertion depending on a precondition', () => {
      22 |    const precondition = setup();
    > 23 |   expect(precondition).toBe(met);
         |             ^
      24 |   const result = fn(precondition);
      25 |   expect(result).toBe(asExpected);
      26 | });

Consider the same case with a hypothetical .soft assertion that continues execution but fails the test case if any (soft or not) assertion is not met

 ● result is as expected given precondition is met

   expect(received).toBe(expected) // Object.is equality

   Expected: met
   Received: notMet

     20 |
     21 | test('an assertion depending on a precondition', () => {
     22 |    const precondition = setup();
   > 23 |   expect.soft(precondition).toBe(met);
        |             ^
     24 |   const result = fn(precondition);
     25 |   expect(result).toBe(asExpected);
     26 | });

   expect(received).toBe(expected) // Object.is equality

   Expected: asExpected
   Received: notAsExpected

     20 |
     21 | test('an assertion depending on a precondition', () => {
     22 |    const precondition = setup();
   > 23 |   expect.soft(precondition).toBe(met);
     24 |   const result = fn(precondition);
     25 |   expect(result).toBe(asExpected);
        |             ^
     26 | });

Pitch

This will lead to better test suites which I take is jest's mission. 🚀

Prior art

Playwright implements this: https://playwright.dev/docs/test-assertions#soft-assertions

[SimenB]

I like the idea! Main issue is that expect is not bound to a test case, so we wouldn't know which test failed (see for instance #8297 (comment)). As mentioned there, and other places, once we bind expect to a single test it'd unblock a bunch of features we'd like. Should probably try to figure out a way to do that at some point...

[liuxingbaoyu]

(function (name) {
    const fn = function () {
        throw 123
    };
    Object.defineProperty(fn, "name", {
        value: name
    })
    fn()
})("'jest_id_xxx'")

This may allow us to use the call stacks.
Very hacky, but compatibility should be fine.
😃

[marionebl]

@SimenB Do you have a rough idea of what binding expect to single tests would entail, is there some sketch of the required work I can read up on?

[alfonso-presa]

It is not to advertise, but to help, but I think this covers the need: https://github.com/alfonso-presa/soft-assert

Let me know if it helps (or not :-) )

[mtaribeiro]

the expect should have the similar functionality than soft-assert.
The idea is minor failures, like text contents, will be displayed and fail only at the end of the test and not stop the test. Therefore the test will validate the functional things.
See the softAssertAll() in https://www.npmjs.com/package/soft-assert