[RFC] More granular default Webpack chunking

[atcastle]

Feature request

Is your feature request related to a problem? Please describe.

The current Webpack chunking strategy in Next.js is based around a ratio-based heuristic for including modules in a single "commons" chunk. Because there is very little granularity, a lot of code is either downloaded unnecessarily (because the commons chunk includes a lot of code that's not actually required for a particular route) or duplicated across multiple page bundles (because it was included in less than half of all pages).

Because these two problems are in opposition, simply adjusting the ratio will always make one better and one worse.

Describe the solution you'd like

I suggest adopting a SplitChunksPlugin configuration such as the following:

cacheGroups: {
    default: false,
    vendors: false,
    framework: {
        name: 'framework',
        chunks: 'all',
        test: /[\\/]node_modules[\\/](react|react-dom)[\\/]/,
        priority: 40
    },
    lib: {
        test(module) {
            return module.size() > 160000
        },
        name(module) {
            return /node_modules\/(.*)/.exec(module.identifier())[1]
                .replace(/\/|\\/g, "_")
        },
        priority: 30,
        minChunks: 1,
        reuseExistingChunk: true
    },
    commons: {
        name: 'commons',
        chunks: 'all',
        minChunks: totalPages,
        priority: 20
    },
    shared: {
	name: false,
        priority: 10,
        minChunks: 2,
        reuseExistingChunk: true
    }               
},
maxInitialRequests: 20

That configuration was designed with these considerations:

Greater granularity

The problems with the current chunking strategy arise from the low granularity of the page-chunk/commons-chunk paradigm. We can easily reduce code duplication by allowing SplitChunksPlugin to create more chunks. In the case of the library shared across 5 entry points, SplitChunksPlugin could simply create a chunk that only contains code shared across those 5 libraries.

Note that SplitChunksPlugin has configuration options to prevent runaway granularity, such as a situation where every module ends up in its own chunk, and the user has to download 100 chunks just to render a route. In particular, in the config above, I'm using the maxInitialRequests option to prevent the number of bundles from skyrocketing.

Cache efficiency

It should be a priority to ensure that shared modules get chunked in a way that causes minimum unnecessary cache invalidation. For an example of this principle, consider an application with 20 entry point chunks. Chunk A and Chunk B both depend on a large library and several small modules. The default SplitChunksPlugin behavior would create a single shared chunk for the intersection of Chunk A and Chunk B, containing the library and modules.

However, in this scenario, all of the following would cache-invalidate the entire chunk, including the large library:

1. Adding a new module depended on by A and B exclusively (adds module to the chunk)
2. Adding a new dependency from a different entry point to one of the modules or the library (module moves out of the chunk)
3. Any change to any of the small modules shared between A and B.

For a sufficiently large library, it makes sense to bundle it all by itself to avoid potential cache invalidations, and the code sample above does exactly that.

A framework chunk

Some modules will be required for every entry point, because they are functionally part of the framework itself, such as React and React-dom. These chunks should be put into a framework chunk, which should be independent from any code introduced by the application developers--even if that code is depended on by 100% of entry points. This is because the frameworks libraries tend to be fairly large, and we know that they generally will not change except when the app developers update the version of Next.js itself. By isolating the framework code we ensure that it will not be cache-invalidated by irrelevant changes made to application code.

Describe alternatives you've considered

I've considered smaller values for maxInitialRequests, but for Next.js applications with a very large number of pages (100+) that can cause a size regression. At 20 that issue goes away. I believe that even at maxInitialRequests 20, the majority of NextJS sites will still have initial requests in the single digits. It's also not clear to me that even if it the initial requests went up to 20 that it would cause any performance degradation for users on modern browsers.

Additional context

In my initial testing on some small and medium sized Next.js apps, this enhanced webpack configuration reduces KB of total JS required for initial page load by as much as 20%. For particularly-poorly optimized sites, I think the improvement could be even greater.

[sokra]

Have you tried using splitChunks.maxSize for granular caching?

[jcruzdzk]

Hi @atcastle , I am trying to split any page for a separated deploy, something like "micro-frontends", you know what I mean? Do you have any suggestion for this?

[Timer]

This is now on-by-default in next@canary. Releasing soon! 🚀

[eltonjothi]

wanted to test out granular chunking but i am having build issues from next@canary,
probably because i am using scss and css (@zeit/next-sass & @zeit/next-css)

Error: Encountered unknown module type: css/extract-chunks. Please open an issue.

[Timer]

@eltonjothi thanks for reporting! Fix is up in #10101.

Edit: should be fixed in next@^9.1.8-canary.16.

[wq93]

Can you provide the code?

[timneutkens]

You can't implement this on older versions of Next.js as it requires a ton of instrumentation. You should upgrade to the latest version, no breaking changes were made between 8 and 9.

Reference #11483

[fabb]

Question: we used to keep unit tests outside of the pages folder to not make them being considered for chunking. Is this obsolete, now that granular chunking has been released?

[timneutkens]

No as Next.js will still turn all js/ts files in pages into entrypoints

[mbogdan0]

Is there a way to disable that granular chunking? I wish I could reduce a number of initial http-requests of my page. Even on a very simple page with no content (plain text, no css, no plugins, anything) I have about a dozen of js chunks and http-requests.

I had to refuse using nextjs due to these reasons.

[timneutkens]

@mbogdan0 you can't disable it, you can read about the reasons behind the way it works here: https://web.dev/granular-chunking-nextjs/

It gives significantly better results than what you're proposing.

[kartikag01]

next.js/packages/next/build/webpack-config.ts

Line 363 in e125d90

test: /(?<!node_modules.*)[\\/]node_modules[\\/](react|react-dom|scheduler|prop-types|use-subscription)[\\/]/,

why next/dist is not a part of the framework chunk?

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.