Improve chunking performance

[lukastaegert]

This PR contains:

• bugfix
• feature
• refactor
• documentation
• other

Are tests included?

• yes (bugfixes and features will not be merged without tests)
• no

Breaking Changes?

• yes (breaking changes will not be merged unless absolutely necessary)
• no, but the output.experimentalDeepDynamicChunkOptimization is deprecated and no longer doing anything and shows a warning instead.

List any relevant issue numbers:

Description

Rollup has an advanced chunking algorithm that can detect if a dependency of a dynamic entry that is shared with the dynamic importer must already be in memory when the dynamic import is loaded. In such a scenario, it will not create a separate chunk for the dependency but import it from the importing chunk.
This can avoid quite a few chunks, but as #4740 showed, the algorithm has a big problem: There is at least a O(e*d*m) complexity with d the number of dynamic imports, e the number of entry points and m the number of modules. And as it turned out for large projects, this could completely blow up.

A stop-gap measure was to introduce the output.experimentalDeepDynamicChunkOptimization option to make the algorithm "dumber" but much faster. This helped performance, but it created quite a few unnecessary chunks for some.

However, I managed to completely rewrite the original algorithm to solve all problems! And part of the solution was to use

BigInt as a high-performance Set replacement!

How does it work? Assume you have a fixed number of objects that you can index with numbers. As BigInts have arbitrary precision, you can then assign each object a bit in the BigInt. So to add element 24, I would do

bigIntSet |= 1n << 24n

Note the single | which is a bitwise OR. But the true power is if I have to compare, merge or intersect such sets:

// comparison: trivial!
bigIntSetA === bigIntSetB

// intersection: trivial, and it is naturally non-mutating!
const intersection = bigIntSetA & bigIntSetB;

// merge: trivial, and again it is naturally non-mutating!
const merged = bigIntSetA | bigIntSetB;

The only thing that is worse for BigInt sets is that you cannot easily iterate Set elements. Either you iterate over all bits, which is unnecessarily many, or one devises some devilish divide-and-conquer scheme that still has for a Set with a single element O(log n) complexity. But luckily, I did not need iteration everywhere.

By restructuring the algorithm make heavy use of such intersections and merges and some other algorithmic improvements, I was able to severely improve performance. Using the example of #4740 as a baseline, I got the following numbers:

#4740 has 1 static entry, 1450 dynamic entries and no manual chunks.

• getChunkAssignments with the current algorithm and experimentalDeepDynamicChunkOptimization disabled:
  • takes 3.9s
  • creates 1874 chunks
• with experimentalDeepDynamicChunkOptimization enabled:
  • takes over 2 hours (!)
  • creates 1742 chunks
• with the new algorithm:
  • takes 3.3s, so faster then the "dumbed down" algorithm. The part that took 2 hours now actually takes only about 600ms!
  • creates 1742 chunks

[netlify]

✅ Deploy Preview for rollupjs ready!

Name	Link
🔨 Latest commit	c1d8b8d
🔍 Latest deploy log	https://app.netlify.com/sites/rollupjs/deploys/63efe93633d25d0008515192
😎 Deploy Preview	https://deploy-preview-4862--rollupjs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

Thank you for your contribution! ❤️

You can try out this pull request locally by installing Rollup via

npm install rollup/rollup#improve-chunking-performance

or load it into the REPL:
https://deploy-preview-4862--rollupjs.netlify.app/repl/?pr=4862

[codecov]

Codecov Report

Merging #4862 (c1d8b8d) into master (ddf09f9) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #4862   +/-   ##
=======================================
  Coverage   98.97%   98.98%           
=======================================
  Files         219      219           
  Lines        7927     7943   +16     
  Branches     2195     2189    -6     
=======================================
+ Hits         7846     7862   +16     
  Misses         26       26           
  Partials       55       55           

Impacted Files	Coverage Δ
src/Bundle.ts	100.00% <100.00%> (ø)
src/Module.ts	100.00% <100.00%> (ø)
src/ast/utils/PathTracker.ts	100.00% <100.00%> (ø)
src/utils/chunkAssignment.ts	100.00% <100.00%> (ø)
src/utils/options/normalizeOutputOptions.ts	100.00% <100.00%> (ø)
src/utils/urls.ts	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[rollup-bot]

This PR has been released as part of rollup@3.17.0. You can test it via npm install rollup.