Report progress of individual test cases

[gaearon]

🚀 Feature Proposal

It seems like Jest currently only reports progress when a complete suite passes.
I propose that instead, it should report progress every second for individual test cases.

Otherwise, the progress output is confusing. "Tests: N" being a separate line in the output gives the impression that it's just as granular as "Test Suites: N" and will be incremented immediately when a test case passes.

Motivation

This is confusing for large/slow suites because you don't get any sense of how much time is left and whether tests are stuck for some reason.

[SimenB]

Sorta related: #6225. But this has an actual use case attached.

[gaearon]

Btw if you want to reproduce, clone facebook/prepack, run yarn and then yarn test-react --watch.

[SimenB]

We don't know the number of tests before executing the test files (and its describes). But populating that information might be possible at that time?

@aaronabramov thoughts?

[gaearon]

I guess my mental model was that each test() or it() call goes into Jest, so Jest should be able to “ping” the watcher about its progress on every exit from an it().

[aaronabramov]

yeah, we should be able to know everything about the number of tests we have once we executed the code in the test file (but before running it)
once we stream the results it should be a one line change to implement that!

[gaearon]

Why is the total number of the tests intersting?

I want to clarify my feature request isn’t about seeing a total number of tests before they run. It’s about seeing the passed number of tests change as each of them completes.

[aaronabramov]

yeah, i agree. we don't need it.
we should just report a single test and increment a counter once it or test is done

[rickhanlonii]

+1, with this I can finish a port of the ava mini reporter to Jest

Here's the current reporter interface:

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  // these are actually for the test suite, not individual test results
  onTestStart(test: Test) {}
  onTestResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

The ideal for me would be (note some of the types are either renamed of do not exist yet):

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}
 
  onTestSuiteStart(suite: TestSuite) {}

  // new: called at the beginning of a .test() or .it() with the test meta info
  onTestStart(test: Test) {}

  // new: called at the end of a .test() or .it() the test meta info and result info
  onTestResult(suite: TestSuite, test: Test, testResult: TestResult, results: AggregatedResult) {}

  onTestSuiteResult(suite: TestSuite, testSuiteResult: TestSuiteResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

Until a major we could call a "test" the test suite and "assertion" an individual test (which pairs with some of the types we use):

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  onTestStart(test: Test) {}
  
  onAssertionStart(assertion: Assertion) {} // note this type doesn't exist

  onAssertionResult(test: Test, assertion: Assertion, assertionResult: AssertionResult, results: AggregatedResult) {}

  onTestSuiteResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

[aaronabramov]

i'm not sure about using the word "assertion" this way.
Right now we have a huge term overload. i think almost everyone at facebook uses "test" to describe the whole file (unlike test suite that we normally use in jest).
then i normally hear the term "test case" to describe individual test or it and assertion to describe an expect or assert call.

[rickhanlonii]

Yeah that makes sense, I like "test case" and that doesn't require renaming anything. I only used "assertion" because the type of a test case result is called AssertionResult

Here's what we can go with then:

export default class BaseReporter {
  // Called at the beginning of a run
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  // Called at the beginning of every test file
  onTestStart(test: Test) {}
  
  // Called at the beginning of every .test or .it
  onTestCaseStart(test: Test, testCase: TestCase) {}

  // Called with the result of every .test or .it
  onTestCaseResult(test: Test, testCase: TestCase, testCaseResult: AssertionResult) {}

  // Called with the result of every test file
  onTestResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  // Called after all tests have completed
  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

[SimenB]

Could we call it Spec instead of TestCase? Matches Jasmine (https://jasmine.github.io/api/3.0/global.html#it).

FWIW the Jest docs calls a single it/test a "test", nowhere I can see is test meant to be the whole file

[gaearon]

nowhere I can see is test meant to be the whole file

👍 I think calling the whole file a "test" is a Facebook-ism, I've never seen this anywhere else. I'd love if Jest stopped using this terminology.

[aaronabramov]

makes sense! if we clean up onTestResult methods and rename them to be onTestSuiteResult we can make it consistent with documentation (and reporters output).
so pretty much what @rickhanlonii wrote earlier:

test_run - all tests/test suites in all runners
test_suite - one file
test - one it/test
assertion - one expect call

[aaronabramov]

hey.. what if we renamed "test suite" to "test file"?
i think this is exactly what it is and it should remove all the confusion.

the conversations i hear all the time (mostly outside fb)

• we have X tests in our codebase
• like.. test suites or actual tests?
• yeah.. suite|tests

[aaronabramov]

@captbaritone you might have some thought on this as well

[thymikee]

I'm actually fot "test file". I often hear a "test suite" referring to all tests in the project, e.g. "We have 100+ tests in our suite."

[captbaritone]

Yes! I mostly agree with @aaronabramov, but I think leaving it/test as "test" is probably a bit ambiguous. Was there a good reason not to use "test case", as @rickhanlonii proposed above?

• "Test Run" - all tests/test suites in all runners
• "Test File" - one file
• "Test Case" - one it/test
• "Assertion" - one expect call

The only thing that this leaves is what to call describe(() => {})s. Maybe it's okay for us not to have a name for this, but if we think it's important to have one, the only thing I can think of is "describe block". If we want to use that name, then maybe an it/test should be a "test block" so that we are consistent.

@SimenB I'm not a fan of "spec" since it seems tightly coupled to the Behavior Driven Development methodology encouraged by Jasmine, and I don't think Jest really prescribes/endorses BDD.

[SimenB]

I still think "test" is best because, well, the function is named test. I could get behind "test case" as well, the only thing I don't like is "test" being the entire file. "Test file" is a good compromise, and it's explicit (which is 👍).

Not sure what to do about describe (or before{All,Each}, after{All,Each} for that matter). "describe block" makes sense, and consistency with "test block" sounds good.

[aaronabramov]

so:

testRun - all files in all runners
testFile - one file
test - one test/it
describeBlock - one describe
assertion - one expect call

i like that!

we already use DescribeBlock in jest-circus https://github.com/facebook/jest/blob/master/types/Circus.js#L182

[captbaritone]

I'm not crazy about leaving "test" as just "test", even though that's the name of the function. There are lots of people, Facebook people included, who have a existing definitions for what "test" means to them. Additionally, for those still using it(), the relationship may not be as obvious.

If we do opt to use "test", then maybe we can just call a describe() a "describe"? That feels consistent with calling a test() a test.

Just my $0.02.

[gaearon]

My choice would be "test case" and "test file".

[SimenB]

I started fiddling with this, and an early snag is that Runner.runTests already takes 6 arguments. Adding more makes it really unwieldy, but changing to passing a single object is a breaking change.

Should we do something about its signature at the same time, or just add more args?

See https://jestjs.io/docs/en/configuration#runner-string

Might make sense to expect runners to be event emitters, then we can just attach listeners instead of passing in callbacks. Could add a property (similar to isSerial) called emitsEvents or something, and fork our logic based on that.

/cc @rogeliog for thoughts on runner api

[hegelstad]

Jest is really lacking on this part.

Is adding custom JasmineReporters really the only option? The documentation on jasmine (in Jest) is really lacking as jest is rolling it's own jasmine bundle.

I propose documenting where users can find valid jasmine documentation for implementing this themselves.

[SimenB]

We're removing the jasmine coupling, so that's no way forward at all

[SimenB]

Looked again at this, and just went with an event emitter as that seemed more sane.

However, I'm having an issue with the test running inside a worker with no way of reporting back to jest until it's done. Any ideas? @mjesun thoughts on a worker posting messages to its parent?

My in progress code, if anyone is interested: https://github.com/SimenB/jest/tree/report-more-events

[SimenB]

This is finally out in 26.2.0! Note that it only works if you're using jest-circus

[SimenB]

Use Jest 26.2.0+ and jest-circus. https://github.com/facebook/jest/blob/master/packages/jest-circus/README.md#installation

[tralston]

Just tried it, and it didn't work.

npm install jest-circus --save-dev
npm update jest

Confirmed that package.json shows jest dependency as "jest": "^26.2.2"

ran npx jest --testRunner='jest-circus/runner'

A particular file I have has ~30 tests, and it showed the same behavior as before. It's not listing the progress of each individual test case.

Am I doing something wrong?

[retorquere]

I'm seeing the same problem as @tralston. I'm still seeing all tests output at once when the last one finishes, and nothing before. I've create an MWE at https://github.com/retorquere/jest-circus-test. Would appreciate feedback on what I should be doing differently.

[SimenB]

@retorquere there's no summary printed when running a single test, see #9420 (comment). Just duplicating your one test fixes the issue.

Not sure it makes sense to hide the summary anymore now since we are able to report stuff as we go

[retorquere]

Then I don't understand what's different now since 26.2.2. In my experience jest has always reported per-file tests this way, even without jest-circus. My use-case is exactly as in my MWE - I generate cases based on sample files, which I now do by pre-generating a test js file per sample, and then run jest, but I'd love to get rid of that.

[tralston]

I guess I misunderstood the purpose of this issue. I never really had a problem with the totals line (e.g. "6 passed, 6 total") being updated as tests passed or failed. I wanted to see (as I thought the title of the issue was reporting) the name and pass/fail of each individual test for each test file, as it's executed. For example, I have a small project with ~50 tests. 40 of them are in a big file, and that's the one we are working on the most. It currently takes ~30 seconds to execute (due to some slow dependencies that eventually I'll mock out). Most of the time I run jest just watching the file I'm working on. It would be really nice to see each test printed out as it's executed, not just the summary updated. Here's an example of what I'm talking about from (Gradle test runner, but with a Mocha theme):

I realize this may not be the right issue for what I'm asking. If not, are you familiar with one like this? Or should I file a new one? I do understand that running tests asynchronously could present some challenges in displaying live results.

[dipasqualew]

@tralston Did you get anywhere with indvidual test reporting while they are being executed? I am looking for the same functionality.

[tralston]

@dipasqualew Unless something changed (@SimenB would know best) this is still a problem for me.

[SimenB]

This issue was as far as I know always about "it feels like nothing is happening" which a live counter fixes. Getting each individual test to show up in the summary as they complete is a separate feature IMO. Feel free to open up an issue or send a PR. The data used for updating the counter should be enough to update the verbose reporter as results come in as well (although I'm not entirely sure how that will work with tests running in parallel - might need to use something like ink)

[dipasqualew]

@SimenB However the live counter only updates when the test case completes and not after each individual test. Do you think that's also out of scope of this issue and part of an extension of the verbose reporter as well?

[SimenB]

Practically all bug reports/feature requests are out of scope in a closed issue. 🙂

It should update after every test, if it does not please open up a separate issue with a reproduction

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.