feat(gatsby): enable granular chunks (#22253)

Enable granular chunking for Gatsby. This PR helps us improve our webpages in 2 ways:

Less duplications (less javascript)
By bundling a dependency that is at least used in 2 pages we can bundle them together so we don't have to download duplicate libraries over and over again. This won't benefit first-page load but it improves page navigation as you'll need less javascript for the next route.

Big libraries over 160kb are moved to a separate library all together to improve js parsing & execution costs.

Our commons chunk is used to put in code that is used over all our pages (commons). So we don't bloat any pages more than we need

Caching
We've added a framework bundle which contains react, react-dom as this is a chunk that will hardly ever change. Commons & shared libraries can change when new pages are created which is more likely to happen than react being upgraded. We might want to move other modules into it, like @reach/router.

Overall caching is improved as chunks will change less often.

docs/docs/production-app.md

@@ -21,7 +21,7 @@ The config is quite large, but here are some of the important values in the fina
 ```javascript
 {
   entry: {
-    app: ".cache/production-app"
+    app: `.cache/production-app`
   },
   output: {
     // e.g. app-2e49587d85e03a033f58.js
@@ -32,6 +32,7 @@ The config is quite large, but here are some of the important values in the fina
     publicPath: `/`
   },
   target: `web`,
+  devtool: `source-map`,
   mode: `production`,
   node: {
     ___filename: true
@@ -41,7 +42,37 @@ The config is quite large, but here are some of the important values in the fina
       // e.g. webpack-runtime-e402cdceeae5fad2aa61.js
       name: `webpack-runtime`
     },
-    splitChunks: false
+    splitChunks: {
+      chunks: `all`,
+      cacheGroups: {
+        // disable Webpack's default cacheGroup
+        default: false,
+        // disable Webpack's default vendor cacheGroup
+        vendors: false,
+        // Create a framework bundle that contains React libraries
+        // They hardly change so we bundle them together to improve
+        framework: {},
+        // Big modules that are over 160kb are moved to their own file to
+        // optimize browser parsing & execution
+        lib: {},
+        // All libraries that are used on all pages are moved into a common chunk
+        commons: {},
+        // When a module is used more than once we create a shared bundle to save user's bandwidth
+        shared: {},
+        // All CSS is bundled into one stylesheet
+        styles: {}
+      },
+      // Keep maximum initial requests to 25
+      maxInitialRequests: 25,
+      // A chunk should be at least 20kb before using splitChunks
+      minSize: 20000
+    },
+    minimizers: [
+      // Minify javascript using Terser (https://terser.org/)
+      plugins.minifyJs(),
+      // Minify CSS by using cssnano (https://cssnano.co/)
+      plugins.minifyCss(),
+    ]
   }
   plugins: [
     // A custom webpack plugin that implements logic to write out chunk-map.json and webpack.stats.json
@@ -52,6 +83,8 @@ The config is quite large, but here are some of the important values in the fina
 
 There's a lot going on here. And this is just a sample of the output that doesn't include the loaders, rules, etc. We won't go over everything here, but most of it is geared towards proper code splitting of your application.
 
+The splitChunks section is the most complex part of the Gatsby webpack config as it configures how Gatsby generates the most optimized bundles for your website. This is referred to as Granular Chunks as Gatsby tries to make the generated JavaScript files as granular as possible by deduplicating all modules. You can read more about [SplitChunks](https://webpack.js.org/plugins/split-chunks-plugin/#optimizationsplitchunks) and [chunks](https://webpack.js.org/concepts/under-the-hood/#chunks) on the [official webpack website](https://webpack.js.org/).
+
 Once Webpack has finished compilation, it will have produced a few key types of bundles:
 
 ##### app-[contenthash].js
@@ -62,6 +95,14 @@ This bundle is produced from [production-app.js](https://github.com/gatsbyjs/gat
 
 This contains the small [webpack-runtime](https://webpack.js.org/concepts/manifest/#runtime) as a separate bundle (configured in `optimization` section). In practice, the app and webpack-runtime are always needed together.
 
+##### framework-[contenthash].js
+
+The framework bundle contains the React framework. Based on user behavior, React hardly gets upgraded to a newer version. Creating a separate bundle improves users' browser cache hit rate as this bundle is likely not going to be updated often.
+
+##### commons-[contenthash].js
+
+Libraries used on every Gatsby page are bundled into the commons javascript file. By bundling these together, you can make sure your users only need to download this bundle once.
+
 ##### component---[name]-[contenthash].js
 
 This is a separate bundle for each page. The mechanics for how these are split off from the main production app are covered in [Code Splitting](/docs/how-code-splitting-works/).
@@ -89,7 +130,7 @@ To show how `production-app` works, let's imagine that you've just refreshed the
 */
 ```
 
-Then, the app, webpack-runtime, component, and data json bundles are loaded via `<link>` and `<script>` (see [HTML tag generation](/docs/html-generation/#5-add-preload-link-and-script-tags)). Now, your `production-app` code starts running.
+Then, the app, webpack-runtime, component, shared libraries, and data json bundles are loaded via `<link>` and `<script>` (see [HTML tag generation](/docs/html-generation/#5-add-preload-link-and-script-tags)). Now, your `production-app` code starts running.
 
 ### onClientEntry (api-runner-browser)
 

packages/gatsby/src/utils/__tests__/webpack-utils.ts

@@ -151,7 +151,7 @@ describe(`webpack utils`, () => {
       it(`includes dependencies that don't use gatsby`, () => {
         expect(
           dependencies.exclude(
-            `/Users/sidharthachatterjee/Code/gatsby-seo-test/node_modules/react/index.js`
+            `/Users/sidharthachatterjee/Code/gatsby-seo-test/node_modules/awesome-lib/index.js`
           )
         ).toEqual(false)
       })

packages/gatsby/src/utils/webpack-utils.ts

@@ -375,8 +375,13 @@ export const createWebpackUtils = (
             ) {
               return true
             }
-            // If dep is babel-runtime or core-js, exclude
-            if (/@babel(?:\/|\\{1,2})runtime|core-js/.test(modulePath)) {
+            // If dep is known library that doesn't need polyfilling, we don't.
+            // TODO this needs rework, this is buggy as hell
+            if (
+              /node_modules[\\/](@babel[\\/]runtime|core-js|react|react-dom|scheduler|prop-types)[\\/]/.test(
+                modulePath
+              )
+            ) {
               return true
             }
 

packages/gatsby/src/utils/webpack.config.js

@@ -1,5 +1,6 @@
 require(`v8-compile-cache`)
 
+const crypto = require(`crypto`)
 const fs = require(`fs-extra`)
 const path = require(`path`)
 const dotenv = require(`dotenv`)
@@ -16,6 +17,8 @@ const apiRunnerNode = require(`./api-runner-node`)
 import { createWebpackUtils } from "./webpack-utils"
 import { hasLocalEslint } from "./local-eslint-config-finder"
 
+const FRAMEWORK_BUNDLES = [`react`, `react-dom`, `scheduler`, `prop-types`]
+
 // Four stages or modes:
 //   1) develop: for `gatsby develop` command, hot reload and CSS injection into page
 //   2) develop-html: same as develop without react-hmre in the babel config for html renderer
@@ -481,6 +484,90 @@ module.exports = async (
 
   if (stage === `build-javascript`) {
     const componentsCount = store.getState().components.size
+    const isCssModule = module => module.type === `css/mini-extract`
+
+    const splitChunks = {
+      chunks: `all`,
+      cacheGroups: {
+        default: false,
+        vendors: false,
+        framework: {
+          chunks: `all`,
+          name: `framework`,
+          // This regex ignores nested copies of framework libraries so they're bundled with their issuer.
+          test: new RegExp(
+            `(?<!node_modules.*)[\\\\/]node_modules[\\\\/](${FRAMEWORK_BUNDLES.join(
+              `|`
+            )})[\\\\/]`
+          ),
+          priority: 40,
+          // Don't let webpack eliminate this chunk (prevents this chunk from becoming a part of the commons chunk)
+          enforce: true,
+        },
+        // if a module is bigger than 160kb from node_modules we make a separate chunk for it
+        lib: {
+          test(module) {
+            return (
+              !isCssModule(module) &&
+              module.size() > 160000 &&
+              /node_modules[/\\]/.test(module.identifier())
+            )
+          },
+          name(module) {
+            const hash = crypto.createHash(`sha1`)
+            if (!module.libIdent) {
+              throw new Error(
+                `Encountered unknown module type: ${module.type}. Please open an issue.`
+              )
+            }
+
+            hash.update(module.libIdent({ context: program.directory }))
+
+            return hash.digest(`hex`).substring(0, 8)
+          },
+          priority: 30,
+          minChunks: 1,
+          reuseExistingChunk: true,
+        },
+        commons: {
+          name: `commons`,
+          // if a chunk is used on all components we put it in commons
+          minChunks: componentsCount,
+          priority: 20,
+        },
+        // If a chunk is used in at least 2 components we create a separate chunk
+        shared: {
+          test(module) {
+            return !isCssModule(module)
+          },
+          name(module, chunks) {
+            const hash = crypto
+              .createHash(`sha1`)
+              .update(chunks.reduce((acc, chunk) => acc + chunk.name, ``))
+              .digest(`hex`)
+
+            return hash
+          },
+          priority: 10,
+          minChunks: 2,
+          reuseExistingChunk: true,
+        },
+
+        // Bundle all css & lazy css into one stylesheet to make sure lazy components do not break
+        // TODO make an exception for css-modules
+        styles: {
+          test(module) {
+            return isCssModule(module)
+          },
+
+          name: `styles`,
+          priority: 40,
+          enforce: true,
+        },
+      },
+      maxInitialRequests: 25,
+      minSize: 20000,
+    }
 
     config.optimization = {
       runtimeChunk: {
@@ -490,39 +577,7 @@ module.exports = async (
       // TODO update to deterministic in webpack 5 (hashed is deprecated)
       // @see https://webpack.js.org/guides/caching/#module-identifiers
       moduleIds: `hashed`,
-      splitChunks: {
-        name: false,
-        chunks: `all`,
-        cacheGroups: {
-          default: false,
-          vendors: false,
-          commons: {
-            name: `commons`,
-            chunks: `all`,
-            // if a chunk is used more than half the components count,
-            // we can assume it's pretty global
-            minChunks: componentsCount > 2 ? componentsCount * 0.5 : 2,
-          },
-          react: {
-            name: `commons`,
-            chunks: `all`,
-            test: /[\\/]node_modules[\\/](react|react-dom|scheduler)[\\/]/,
-          },
-          // Only create one CSS file to avoid
-          // problems with code-split CSS loading in different orders
-          // causing inconsistent/non-determanistic styling
-          // See https://github.com/gatsbyjs/gatsby/issues/11072
-          styles: {
-            name: `styles`,
-            // This should cover all our types of CSS.
-            test: /\.(css|scss|sass|less|styl)$/,
-            chunks: `all`,
-            enforce: true,
-            // this rule trumps all other rules because of the priority.
-            priority: 10,
-          },
-        },
-      },
+      splitChunks,
       minimizer: [
         // TODO: maybe this option should be noMinimize?
         !program.noUglify &&