[Question] Tree shaking

[iamursky]

@evanw, are you planning to add tree shaking to exclude unused code from bundles?

[evanw]

Yes, I'm planning on doing this, but it'll likely be after other features land first.

[evanw]

Note to self: There's a tree shaking size comparison here that could be interesting: #86 (comment).

[kzc]

Another example: #81 (comment)

[evanw]

Thanks for these links! These are very helpful.

I have tree shaking basically implemented, and about to be released. It's currently in a branch as I test it and prepare it for release. This release is going to be about correctness, not minimal bundle sizes. It introduces the tree shaking data structure but doesn't yet handle the side-effect annotations you mention in #86 (comment). That will have to come in subsequent releases.

Is there information anywhere about how the "sideEffects" hint is supposed to work? Using it to remove the whole module if it's unreferenced makes sense to me. I'm wondering if it's supposed to be used to infer something about the top-level statements in the file not having side-effects too.

[kzc]

Just as one would find the defacto pure annotations test examples in the uglify-js tree where it was first implemented, you'd have to look at the webpack source tree for "sideEffects" test cases. The last link in #86 (comment) was actually the official documentation for the latter.

[evanw]

Yeah I'll have to study the test cases, and do some reverse engineering to answer some of my questions. I read the documentation but it was too high-level and didn't really tell me enough information to implement it.

I just released version 0.4.0 which has the preliminary tree shaking support described above. Please try it out and let me know if you encounter any correctness issues. As I said above, esbuild isn't respecting annotations yet so the size is likely still going to be bigger than other bundlers. Right now tree shaking is only based on which top-level statements reference which other top-level statements. I'll do another push later on to improve on bundle sizes.

[kzc]

The "sideEffects" package hint is relatively simple - it is either true, false, or an array of files within the module with side effects - implying that all files not listed in the array are considered to be side-effect-free (whether actually the case or not). Here's an array example:

https://github.com/webpack/webpack/blob/69f03be1ea5187e681d3295e679789aa0e6364a5/test/cases/side-effects/order-issue-7665/module/package.json

"sideEffects": true and "sideEffects": [] are effectively the same - all files within the package are considered to have side effects.

If a side effect free file within a module is imported and no symbols are used from it, then the import can be dropped altogether. But if any exported symbol is imported from a side effect free file then its top level statements with side effects must be retained, and only its unreferenced symbols without side effects may be dropped. To use a garbage collector analogy, any used exports or top level statements with side effects are considered to be roots.

Here's an example with additional documentation: https://github.com/webpack/webpack/tree/master/examples/side-effects

[kzc]

The following was observed behavior from rollup's implementation:

"sideEffects": true and "sideEffects": [] are effectively the same - all files within the package are considered to have side effects.

but it doesn't seem to be logically consistent with
"sideEffects": [ "file_not_present_in_package.js" ]
which makes the entire package behave like "sideEffects": false.

I don't know whether webpack has the same behavior as rollup with an empty "sideEffects" array. If they are in disagreement, use webpack's behavior.

[floydspace]

🎉🎉🎉
Preliminary tree shaking support
Great job @evanw

Upgrading serverless-esbuild

[evanw]

If a side effect free file within a module is imported and no symbols are used from it, then the import can be dropped altogether.

Yes, that model makes sense to me. That's the trivial way to do things safely.

To use a garbage collector analogy, any used exports or top level statements with side effects are considered to be roots.

This is what I've implemented in the latest release.

But if any exported symbol is imported from a side effect free file then its top level statements with side effects must be retained, and only its unreferenced symbols without side effects may be dropped.

Since code is JavaScript, it's impossible to determine what is guaranteed to be free of side effects outside of a whitelist of a few types of expressions such as literals. I was wondering if the "sideEffects": false annotation somehow biases the compiler's determination of what is side-effect free. For example, you might have code like this:

export function foo() {}
foo.bar = 123

It's impossible to determine based on this code whether the second statement has side effects or not. Someone might have done this somewhere before this was evaluated:

Object.defineProperty(Function.prototype, 'bar', {
  set() {
    sideEffects()
  }
})

And if the second statement is retained then the first line must also be retained. I'm wondering if "sideEffects": false is supposed to mean something like "trust me these property accesses don't have side effects". Otherwise tree shaking isn't going to do that much in the benchmarks you referenced.

I could see an alternative algorithm where, if "sideEffects": false is present, code with side-effects can form a connected component in the top-level statement graph but connected components could still be dropped if there are no references into the connected component from the outside (also similar to garbage collection). But referencing something in the connected component would pull in the whole connected component. So referencing foo would also pull in the assignment to foo.bar.

[kzc]

I was wondering if the "sideEffects": false annotation somehow biases the compiler's determination of what is side-effect free.

The side effect determination ought be the same whether or not "sideEffects": false is in effect and an export from a given file is used.

Be aware that Rollup and Terser can be fooled by pathological cases of Object.defineProperty pretty easily even with normal code without a "sideEffects" package hint. See also: rollup/rollup#2219.

[evanw]

For people following along, there's a thread relevant to tree shaking in #86. Let's merge that thread into this one.

I have cloned https://github.com/mischnic/tree-shaking-example into https://github.com/evanw/tree-shaking-example and added support for esbuild. I also integrated the ideas around correctness verification from #86 and attempted to fix some of the correctness issues with Parcel and Rollup. Bundles that behave incorrectly are invalid, and their sizes no longer count. There are still some Parcel and Rollup bugs that I couldn't fix.

I just released esbuild 0.4.6 with support for sideEffects. Here is the latest run of those benchmarks including all esbuild tree shaking progress so far:

file                       size      error
-------------------------  --------  --------------------------------------
rollup/lodash-es             18.3kb
parcel/lodash-es             18.8kb
webpack/lodash-es            20.6kb
esbuild-0.4.6/lodash-es      21.3kb
esbuild-0.4.2/lodash-es      91.4kb
esbuild-0.3.9/lodash-es      92.0kb

rollup/lodash                69.2kb
esbuild-0.3.9/lodash         70.4kb
esbuild-0.4.2/lodash         70.4kb
esbuild-0.4.6/lodash         70.4kb
webpack/lodash               72.2kb
parcel/lodash              !!!!!!!!  Attempted to require "buffer"

rollup/rxjs                   9.3kb
parcel/rxjs                   9.8kb
webpack/rxjs                 11.0kb
esbuild-0.4.2/rxjs           86.8kb
esbuild-0.4.6/rxjs           86.8kb
esbuild-0.3.9/rxjs          113.2kb

parcel/react-icons            9.6kb
rollup/react-icons            9.7kb
webpack/react-icons          10.2kb
esbuild-0.4.2/react-icons  1241.9kb
esbuild-0.4.6/react-icons  1241.9kb
esbuild-0.3.9/react-icons  !!!!!!!!  Missing file: react-icons.js

rollup/remeda                 2.0kb
parcel/remeda                 2.1kb
webpack/remeda                2.9kb
esbuild-0.4.2/remeda          7.6kb
esbuild-0.4.6/remeda          7.6kb
esbuild-0.3.9/remeda         13.0kb

rollup/ramda                  6.3kb
parcel/ramda                  6.5kb
webpack/ramda                 7.3kb
esbuild-0.4.2/ramda          44.7kb
esbuild-0.4.6/ramda          44.7kb
esbuild-0.3.9/ramda          47.7kb

parcel/ramdaBabel             6.9kb
esbuild-0.4.2/ramdaBabel      7.8kb
esbuild-0.4.6/ramdaBabel      7.8kb
esbuild-0.3.9/ramdaBabel      7.9kb
webpack/ramdaBabel            8.5kb
rollup/ramdaBabel          !!!!!!!!  Attempted to require "ramda/src/range"

rollup/rambda                 0.8kb
esbuild-0.4.2/rambda          1.8kb
esbuild-0.4.6/rambda          1.8kb
webpack/rambda                2.5kb
parcel/rambda                 9.8kb
esbuild-0.3.9/rambda         12.1kb

rollup/rambdax                6.2kb
esbuild-0.4.2/rambdax         7.8kb
esbuild-0.4.6/rambdax         7.8kb
webpack/rambdax               8.4kb
parcel/rambdax               22.5kb
esbuild-0.3.9/rambdax        34.9kb

The jump from 0.3.9 to 0.4.2 is the initial tree shaking support for ES6 imports and exports. This improves various benchmarks including rxjs (113.2kb => 86.8kb) and rambda (12.1kb => 1.8kb). This also fixes a bug with re-exports in react-icons.

The jump from 0.4.2 to 0.4.6 is the newly-released support for the sideEffects hint. This improves the lodash-es benchmark (91.4kb => 21.3kb) but doesn't affect the other benchmarks at all.

Follow-up work is support for __PURE__ hints, which are not supported yet. Big thanks to @kzc for all of the help so far.

[kzc]

Please update to the latest versions of parcel, rollup, webpack, babel and related packages:

--- a/package.json
+++ b/package.json
@@ -20,22 +20,21 @@
                "rxjs": "^6.5.2"
        },
        "devDependencies": {
-               "@babel/core": "^7.4.4",
-               "@babel/preset-env": "^7.4.4",
-               "babel-loader": "^8.0.6",
+               "@babel/core": "7.10.2",
+               "@babel/preset-env": "7.10.1",
+               "babel-loader": "8.1.0",
                "console.table": "^0.10.0",
                "filesize": "^4.1.2",
                "npm-run-all": "^4.1.5",
-               "nyc": "^14.1.1",
-               "parcel-bundler": "^1.12.3",
-               "rollup": "^1.12.1",
-               "rollup-plugin-babel": "^4.3.2",
-               "rollup-plugin-commonjs": "^10.0.0",
-               "rollup-plugin-node-resolve": "^5.0.0",
-               "rollup-plugin-terser": "^4.0.4",
-               "terser-webpack-plugin": "^1.2.4",
-               "webpack": "^4.31.0",
-               "webpack-cli": "^3.3.2"
+               "parcel-bundler": "1.12.4",
+               "rollup": "2.13.1",
+               "rollup-plugin-babel": "4.4.0",
+               "rollup-pluginutils": "2.8.2",
+               "rollup-plugin-commonjs": "10.1.0",
+               "rollup-plugin-node-resolve": "5.2.0",
+               "rollup-plugin-terser": "6.1.0",
+               "webpack": "4.43.0",
+               "webpack-cli": "3.3.11"
        },
        "scripts": {
                "clean": "rm -rf .cache parcel rollup webpack esbuild-*",
@@ -55,7 +54,7 @@
                "esbuild:rambda": "LIB='rambda' node esbuild.js",
                "esbuild:ramda": "LIB='ramda' node esbuild.js",
                "esbuild:ramdaBabel": "LIB='ramdaBabel' node esbuild.js",
-               "parcel:lodash": "parcel build src/lodash.js --experimental-scope-hoisting -d parcel",
+               "parcel:lodash": "parcel build src/lodash.js -d parcel",
                "parcel:lodash-es": "parcel build src/lodash-es.js --experimental-scope-hoisting -d parcel",
                "parcel:rxjs": "parcel build src/rxjs.js --experimental-scope-hoisting -d parcel",
                "parcel:react-icons": "parcel build src/react-icons.js --experimental-scope-hoisting -d parcel",

rollup@2.x sourcemap configuration changed in the last major. This is the simplest workaround:

--- a/rollup.config.js
+++ b/rollup.config.js
@@ -27,5 +27,4 @@ export default [
                                        }
                                },
-                               sourcemap: false,
                                toplevel: true
                        })

parcel/lodash !!!!!!!! Attempted to require "buffer"

Might be a parcel scope hoisting bug. Disabling --experimental-scope-hoisting works for that test. See package.json diff above.

rollup/lodash                70.6kb                              
webpack/lodash               72.2kb                              
parcel/lodash                92.6kb                              

@mischnic could provide a better answer.

rollup/ramdaBabel !!!!!!!! Attempted to require "ramda/src/range"

CJS require/exports and ES import/export constructs should not be mixed within the same source file. This works for all bundlers:

--- a/src/ramdaBabel.js
+++ b/src/ramdaBabel.js
@@ -13,4 +13,4 @@ function fn(x) {
        )(x);
 }
 
-export const answer = fn(10).join(',');
+exports.answer = fn(10).join(',');

rollup/ramdaBabel             6.5kb                              
parcel/ramdaBabel             6.9kb                              
webpack/ramdaBabel            8.5kb