Show progress of the addons-storyshot run

[markelog]

Bug or support request summary

When executing screenshot tests with addons-storyshot plugin on the large project it shows only initial start message then it looks like it froze when in reality it just has a lot things to do. I'm wondering it it's all possible to show the progress of it?

Steps to reproduce

I suppose default setup of addons-storyshot will do

Please specify which version of Storybook and optionally any affected addons that you're running

• @storybook/react@3.2.12
• @storybook/addon-storyshots@4.0.0-alpha.6

[Keraito]

I tried my hands a little bit onto this, but any attempts in trying to log something to the terminal from https://github.com/storybooks/storybook/blob/master/addons/storyshots/src/index.js when running the tests failed for me. This seem to involve some irregular longstanding issue at Jest regarding console output being swallowed by the tests output. Maybe @ndelangen you have a bit more insight into a way for this?

[ndelangen]

We should probably reach out the the jest team and ask them if they'd have some way in mind to get this. I know we're already kinda doing things out of the ordinary.

Maybe @orta or @cpojer or @SimenB could help here?

[SimenB]

I have no idea how storyshots is implemented under the hood beyond a cursory look now, but I think some way of making separate story files into separate test files would be the best. We collect all results from a single test file before passing the result on to reporters. I don't think we have a way of streaming the results per test.

That would also help keep the .snap files smaller, and utilize parallelization.

Not sure how feasible that is, though.

[ndelangen]

Hmm I'm guessing a totally alternative route that would make that work, would be to add story-files as test-files themselves and have a transformer applied on them?

But that sounds easier then it is, since storybook support global options and decorators..
Still.. not impossible.

[SimenB]

FWIW we've got an issue for reporting tests as completed as they are, rather than wait for the whole suite to complete (jestjs/jest#6616).

I'd still recommend looking into splitting it up into multiple test files for better concurrency.

[Keraito]

Re-opening this issue based on the previous comment as the related topic is atm being discussed in Jest itself.

I'd still recommend looking into splitting it up into multiple test files for better concurrency.

@ndelangen would it make sense to make a note about this in the storyshots documentation? Or hold of to see what happens at Jest's side?

[igor-dv]

BTW, with the new feature, It might be much easier

[ndelangen]

Interesting @igor-dv !

The splitting into files you mean?
Yeah I can see how.

Requires a bulk of work on the user's part that way.

If we could leverage the component-story files themselves, people would not have to add any files, in fact remove some..

[ndelangen]

On the roadmap:
https://storybook.canny.io/roadmap/p/storyshots-continues-feedback-during-run

[SimenB]

Nice! I wonder if that'll fix/help #3286 at the same time

[ndelangen]

Yeah possibly

[verticalgrain]

Update in 2019, one can do this:

const beforeScreenshot = ( page, { context: { kind, story }, url } ) => {
	console.log( 'Generating screenshot for: ' + story );
	return new Promise( resolve =>
		setTimeout( () => {
			resolve();
		}, 2000 )
	)
}

[SQReder]

@verticalgrain I did the same, but without delay

[SimenB]

If storyshots is already separate test cases within a test file (which I believe it is: https://github.com/storybookjs/storybook/blob/0cae839ac299edcdc5e69116913835a769173f1f/addons/storyshots/storyshots-core/src/api/snapshotsTestsTemplate.ts), this should be fixed in Jest 26.3+ if you use jest-circus (which will be the default in Jest 27).

Again, separate files for tests are probably better in order to properly utilize parallelization across CPUs, but the originally reported "no feedback until complete" should be solved.

[ndelangen]

@SimenB that's awesome!

related: #4996

[bard]

Again, separate files for tests are probably better in order to properly utilize parallelization across CPUs, but the originally reported "no feedback until complete" should be solved.

@SimenB is there any way to feed Jest "virtual" (i.e. not written to disk) files generated on the fly? That way one could generate one test file per story or one file per story kind. Parallelization aside, I'm interested in the possibility of relating those test files back to their respective story files via import, so in theory Jest's --findRelatedFiles could kick in, and only tests for modified components would run.

[SimenB]

@bard no, Jest currently only works on a real FS. Not sure if it's worth abstracting out the file system entirely (e.g. lots of Jest's internal modules are used outside of Jest, so the API might get hairy). That said, we've been wanting to virtualize the FS previously to make it possible to run Jest in the browser: jestjs/jest#848. That has stagnated, though

[ndelangen]

The future of storyshots is the test-runner:
https://storybook.js.org/docs/react/writing-tests/test-runner#page-top

And using the play function for asserts:
https://storybook.js.org/docs/react/writing-stories/play-function#page-top

We will not be making significant improvements/changes to storyshots.