Very slow builds with many files

[selbekk]

Current Behavior

When building a single entry file with ~160 exports (each in their own file), the build process takes several minutes without a warm cache (crashes with an OOM error on CI)

Expected behavior

I'd expect it to be slower than a single module file, but not THAT much slower

Suggested solution(s)

It looks like it's a issue with either rollup or one of its plugins - but I have no way of knowing. Perhaps you've stumbled upon this issue previously?

Also, perhaps a --perf flag could be added to the CLI, to simplify this debugging process for future users?

Additional context

I added a { perf: true; } flag to the rollup config, and console.log-ed out the resulting getTimings() call:

Here's the output

{ '# BUILD': [ 45448.705339, 242397656, 537384520 ],
  '## parse modules': [ 90855.13704900001, 489351016, 541983792 ],
  '- plugin 0 (node-resolve) - resolveId': [ 16.686786000000005, 1722304, 334375352 ],
  '- plugin 0 (node-resolve) - resolveId (async)': [ 87850.61065700013, 4134695144, 346438824 ],
  'load modules': [ 1401519.6063479998, 1050694920, 352076056 ],
  '- plugin 0 (node-resolve) - load': [ 11.477684000000005, 94360, 346416168 ],
  '- plugin 3 (rpt2) - load': [ 11.088360000000012, 104168, 346420296 ],
  '- plugin 4 (babel) - load': [ 11.090018000000006, 89096, 346423304 ],
  '- plugin 6 (sourcemaps) - load': [ 16.119847000000004, 647744, 346432968 ],
  '- plugin 6 (sourcemaps) - load (async)': [ 943857.0503409998, 615032768, 352005520 ],
  '- plugin 5 (sourcemaps) - load': [ 9.865984000000001, 326648, 346426376 ],
  '- plugin 5 (sourcemaps) - load (async)': [ 452015.570979, 316842608, 352004560 ],
  '- plugin 1 (json) - transform': [ 12.718466, 76960, 352309288 ],
  '- plugin 2 (at position 3) - transform': [ 13.978003999999999, 144240, 352517584 ],
  '- plugin 3 (rpt2) - transform': [ 38510.69158600002, 247762048, 592937568 ],
  '- plugin 4 (babel) - transform': [ 1020.9119190000005, 24824640, 537346512 ],
  '- plugin 4 (babel) - transform (async)': [ 513363.3542649998, 25696740784, 581710376 ],
  '- plugin 5 (replace) - transform': [ 55.411102000000014, 4937944, 589553936 ],
  'generate ast': [ 223.65392300000016, 10677648, 593264432 ],
  'analyse ast': [ 185.6019320000002, -246277192, 593236144 ],
  '- plugin 3 (rpt2) - resolveId': [ 91.53104199999997, -5452616, 346475520 ],
  '## analyse dependency graph': [ 23.803683, 4306704, 544409184 ],
  '## mark included statements': [ 22.00839, -6872496, 540410416 ],
  'treeshaking pass 1': [ 7.548756, 1165800, 545589000 ],
  'treeshaking pass 2': [ 9.726016, -8792840, 540409800 ],
  '## generate chunks': [ 10.429739, 1763976, 541256168 ] }
{ '# BUILD': [ 136458.975271, 742819144, 545222456 ],
  '## parse modules': [ 136328.120818, 735788816, 541983792 ],
  '- plugin 0 (node-resolve) - resolveId': [ 16.686786000000005, 1722304, 334375352 ],
  '- plugin 0 (node-resolve) - resolveId (async)': [ 87850.61065700013, 4134695144, 346438824 ],
  'load modules': [ 1401519.6063479998, 1050694920, 352076056 ],
  '- plugin 0 (node-resolve) - load': [ 11.477684000000005, 94360, 346416168 ],
  '- plugin 3 (rpt2) - load': [ 11.088360000000012, 104168, 346420296 ],
  '- plugin 4 (babel) - load': [ 11.090018000000006, 89096, 346423304 ],
  '- plugin 6 (sourcemaps) - load': [ 16.119847000000004, 647744, 346432968 ],
  '- plugin 6 (sourcemaps) - load (async)': [ 943857.0503409998, 615032768, 352005520 ],
  '- plugin 5 (sourcemaps) - load': [ 9.865984000000001, 326648, 346426376 ],
  '- plugin 5 (sourcemaps) - load (async)': [ 452015.570979, 316842608, 352004560 ],
  '- plugin 1 (json) - transform': [ 12.718466, 76960, 352309288 ],
  '- plugin 2 (at position 3) - transform': [ 13.978003999999999, 144240, 352517584 ],
  '- plugin 3 (rpt2) - transform': [ 38510.69158600002, 247762048, 592937568 ],
  '- plugin 4 (babel) - transform': [ 1020.9119190000005, 24824640, 537346512 ],
  '- plugin 4 (babel) - transform (async)': [ 513363.3542649998, 25696740784, 581710376 ],
  '- plugin 5 (replace) - transform': [ 55.411102000000014, 4937944, 589553936 ],
  'generate ast': [ 223.65392300000016, 10677648, 593264432 ],
  'analyse ast': [ 185.6019320000002, -246277192, 593236144 ],
  '- plugin 3 (rpt2) - resolveId': [ 91.53104199999997, -5452616, 346475520 ],
  '## analyse dependency graph': [ 37.019328, 6046336, 544409184 ],
  '## mark included statements': [ 32.907793999999996, -5799072, 544293544 ],
  'treeshaking pass 1': [ 13.899056, 1559440, 545589000 ],
  'treeshaking pass 2': [ 11.831361, -8455048, 544293088 ],
  '## generate chunks': [ 14.988824999999999, 2623968, 545153640 ],
  '# GENERATE': [ 0, 0, 0 ] }
{ '# BUILD': [ 136458.975271, 742819144, 545222456 ],
  '## parse modules': [ 136328.120818, 735788816, 541983792 ],
  '- plugin 0 (node-resolve) - resolveId': [ 16.686786000000005, 1722304, 334375352 ],
  '- plugin 0 (node-resolve) - resolveId (async)': [ 87850.61065700013, 4134695144, 346438824 ],
  'load modules': [ 1401519.6063479998, 1050694920, 352076056 ],
  '- plugin 0 (node-resolve) - load': [ 11.477684000000005, 94360, 346416168 ],
  '- plugin 3 (rpt2) - load': [ 11.088360000000012, 104168, 346420296 ],
  '- plugin 4 (babel) - load': [ 11.090018000000006, 89096, 346423304 ],
  '- plugin 6 (sourcemaps) - load': [ 16.119847000000004, 647744, 346432968 ],
  '- plugin 6 (sourcemaps) - load (async)': [ 943857.0503409998, 615032768, 352005520 ],
  '- plugin 5 (sourcemaps) - load': [ 9.865984000000001, 326648, 346426376 ],
  '- plugin 5 (sourcemaps) - load (async)': [ 452015.570979, 316842608, 352004560 ],
  '- plugin 1 (json) - transform': [ 12.718466, 76960, 352309288 ],
  '- plugin 2 (at position 3) - transform': [ 13.978003999999999, 144240, 352517584 ],
  '- plugin 3 (rpt2) - transform': [ 38510.69158600002, 247762048, 592937568 ],
  '- plugin 4 (babel) - transform': [ 1020.9119190000005, 24824640, 537346512 ],
  '- plugin 4 (babel) - transform (async)': [ 513363.3542649998, 25696740784, 581710376 ],
  '- plugin 5 (replace) - transform': [ 55.411102000000014, 4937944, 589553936 ],
  'generate ast': [ 223.65392300000016, 10677648, 593264432 ],
  'analyse ast': [ 185.6019320000002, -246277192, 593236144 ],
  '- plugin 3 (rpt2) - resolveId': [ 91.53104199999997, -5452616, 346475520 ],
  '## analyse dependency graph': [ 37.019328, 6046336, 544409184 ],
  '## mark included statements': [ 32.907793999999996, -5799072, 544293544 ],
  'treeshaking pass 1': [ 13.899056, 1559440, 545589000 ],
  'treeshaking pass 2': [ 11.831361, -8455048, 544293088 ],
  '## generate chunks': [ 14.988824999999999, 2623968, 545153640 ],

The "Building modules" stage of the build process takes 47.3 seconds - that's for building the 160 SVGR generated React components (it's an icon library) on my 2.7GHz i7 16GB RAM MacBook Pro.

Your environment

Software	Version(s)
TSDX	0.9.0
TypeScript	3.6.2
Browser	n/a
npm/Yarn	yarn 1.17.3
Operating System	osx 10.14.6

[swyxio]

instead of a --perf flag we should use https://www.npmjs.com/package/debug. happy to take a PR if you're interested in working on that

[selbekk]

Hm I guess I could look at it more closely next week. Haven't used debug yet though

[swyxio]

its really good. basically gold standard for logging in cli's.

[selbekk]

Cool stuff. I'll look into it for sure - but what would you expect from a PR? I'd love to use this as a learning opportunity :) hey, perhaps i'll turn it into something fun for this year's react.christmas? 😅

[swyxio]

you can implement just the logging that you need to dig into your perf issues. once we understand the source of your perf issues then we can make a separate PR to fix that. make sense?

[selbekk]

that'll work :)

[selbekk]

I solved this by using the rollup plugin for svgr instead (per #200), but I'll still have a look at the debugging part.

Closing this issue as I won't be working on fixing the "many files, slow build" issue.

[agilgur5]

This was also worked on in #243

[coreybruyere]

@selbekk - you mind sharing you tsdx rollup config? I'm using the default tsdx config and am getting terribly slow build times.

[selbekk]

I'm no longer working on that particular project, and has lost access to the source code unfortunately.