Jest Web Test Run Initialization is Slow O(10x) Slower than it Should Be

[MichaelrMentele]

Problem When running jest tests the runtime seems to be 12-14 seconds just to spin up a test and mount a component. The actual tests are fast. That implies 99% of the wall time is due to setup -- possibly because we have several chained yarn calls but most likely due to source maps (usual suspect for this).

As evidence, API tests are MUCH faster.

 PASS   api  api/src/services/stages/stages.test.ts
  stages
    ✓ returns all stages (45 ms)
    ✓ returns a single stage (19 ms)
    ✓ creates a stage (16 ms)
    ✓ updates a stage (14 ms)
    ✓ deletes a stage (12 ms)

Test Suites: 1 passed, 1 total
Tests:       5 passed, 5 total
Snapshots:   0 total
Time:        1.043 s

 FAIL   web  web/src/layouts/DefaultLayout/DefaultLayout.test.tsx (13.13 s)
  DefaultLayout
    ✓ defaults to open (260 ms)
    ✓ opens on menu click (88 ms)
    ✓ closes on chevron click (58 ms)
    ✕ should navigate to '/home' (47 ms)
    ✓ renders successfully (33 ms)

Resolution
We should probably add an option to ignore source maps / use an ugly version by default and then use a full fidelity one when useing a debugger launch config -- something like that.

I have a PR cooking that adds the debugging config for jest in vscode. I may be able to address this as well with a bit of advice on where to stick an option -- perhaps just as an ENV variable.

[thedavidprice]

Looping in @dac09 and @callingmedic911

Related to #4569 #4360 — not necessarily duplicate but we definitely want to keep this topic linked/consolidated as much as possible.

[callingmedic911]

Thanks for working on it @MichaelrMentele!

I will defer to @dac09 on how/if we can configure this. Meanwhile, would it be possible to share the difference in the benchmark with and without the sourcemap?

[MichaelrMentele]

Let me take a poke at it.

[MichaelrMentele]

Should get to it this week

[MichaelrMentele]

Feeling a bit blocking profiling this by #5425

[MichaelrMentele]

I did toggle the source map in a generated webpack config -- I need to run several trials to see if it made a reasonable difference.

[MichaelrMentele]

I noted that the actual subtests take O(ms) while the net runtime takes O(10s) this implies that the bottleneck is at transpilation or loading or some other setup / initialization step. I had a vague idea that it might have to do with building source maps.

However, it looks like source maps was NOT the bottleneck:

• removing source maps made about a 5% difference (it seems) in run time for tests

Another idea is to use runInBand since then the setup would be shared (the transpilation at least) and indeed this seemed about 3x faster on an average test basis but slower on a total wait time basis.

It's difficult to actually profile the jest test -- other than simply using a node profiler which is awkward to actually pass down through the redwood CLI (not quite sure how to do it)

I started looking at how things were cached. My understanding is that jest relies on babel directly to transpile code and doesn't use webpack for bundling at all (other than for transforming assets perhaps which we don't care about for headless jest tests anyway).

A few areas that are promising:

• babel-loader cacheDirectory -- we aren't actually caching the transpiled code anywhere I can see for webpack builds with babel
• mocking out assets https://jestjs.io/docs/webpack#:~:text=Jest%20can%20be%20used%20in,assets%2C%20styles%2C%20and%20compilation.
• making sure we only include source files

Interesting reference for ideas:

• Slow startup time in monorepos (single test/project) jestjs/jest#10833
• Very poor performance on basic tests jestjs/jest#7963

[finalight]

are there plans to use swc/jest? they could significantly increase the speed of the test itself

[dac09]

Hey @MichaelrMentele - I wonder if you've had a chance to try out your project on Redwood V3? Hoping that this improves the performance for you.

are there plans to use swc/jest?

Hey @finalight - thanks for the suggestion here - but I'm not super sure we would see much benefit from swc - see this issue here: swc-project/swc#1172 and in particular a comment that really resonated

For testing, especially benchmarks, transpiling code is likely less than 1% of the time spent on the test, so optimising 1% of a process will lead to the results you are seeing.

We would love to move in that direction in general, but we'd have to move the custom babel plugins (see this) we use to SWC plugins (which are in Rust). Would be great to start in that direction, but its outside of my comfort zone - maybe something you would be interested in?

SWC also currently has a big limitation right now (AFAIK) - the polyfill process swc-project/swc#5584 - which means internally its not something we can use to build Redwood packages right now.

[MichaelrMentele]

I do believe the initialization is much improved to your optimizations 🎉 🎉 @dac09

Still more startup time than I would expect. On the API side it still takes several seconds to start a test e.g.

 PASS   api  api/src/services/residences/residences.test.ts (5.03 s)
  residences
    ✓ returns all residences (403 ms)
    ✓ returns a single residence (102 ms)
    ✓ creates a residence (106 ms)
    ✓ updates a residence (99 ms)
    ✓ deletes a residence (98 ms)

[dac09]

Hey Michael, thanks for getting back and confirming things are better for you!

I'll close this down, but know that we do have an initiative to see if we can further improve the performance of scenarios!