Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack: strictly verify imports and disable nodejs polyfilling #22630

Merged
merged 14 commits into from
Mar 2, 2018

Conversation

samouri
Copy link
Contributor

@samouri samouri commented Feb 20, 2018

As discussed in an issue earlier, the best way to make sure we keep our bundled code size lean is to make ensure we stop it from growing. This PR focuses on that goal by modifying how we interact with the nodejs modules.

nodejs has a standard library that does not exist in browsers. this dichotomy is confusing to develop in, and in order to alleviate some of the resulting frustration webpack automatically polyfills many of these libs. The problem: to someone that doesn't know, these libs appear as if they are just freely available but actually they don't come free, and can easily cause bloat. We should include them with the same scrutiny that we use when including any other npm dependency.

In order to fix this going forward there were two changes I needed to make:

  1. disable nodejs polyfilling within webpack
  2. eslint rules so that we only depend on things explicitly declared within package.json

This could have saved us in a few cases like:

  1. Framework: Substitute crypto polyfill dependants with smaller stand-ins #17356
  2. Rewrite store-transactions submit to not use Readable #22539

notes as i've been going along:

  • the jitm data-layer uses the variable process. I confirmed locally that shadowing the variable still works even though we are overwriting process with DefinePlugin.
  • the wpcom lib uses fs in certain conditions which is invalid in the browsers. This is why i've added a generic mock function in its place. Feel free to be critical of how I handled this bit because it feels like there could be a better way.
  • libraries often try to access thing in the global scope for things like feature-detection. for example uuid just calls crypto. I don't know why, but webpack doesn't allow for that explicitly, which is why I needed set global to {} with DefinePlugin
  • cachingActionCreatorFactory uses lru-cache, which isn't even a declared dependency in package.json. The only reason Calypso is compiling is because it is a dependency of a dependency. It was written for node and node for the browser. I'll replace it with lru.

To test
add these snippets to any file in client and make sure it crashes the build:

const crypto = require('crypto');
const cert1 = new crypto.Certificate();
import http from 'http';
http.get({
  hostname: 'localhost',
  port: 80,
  path: '/',
})

@matticbot
Copy link
Contributor

matticbot commented Feb 20, 2018

@samouri samouri self-assigned this Feb 20, 2018
@samouri samouri changed the base branch from master to remove/stream-browserify February 20, 2018 19:16
@jsnajdr
Copy link
Member

jsnajdr commented Feb 20, 2018

a wpcom lib uses fs somewhere. need to dig deeper

I noticed that too when working on the REST Proxy 2.0. The library can run also in Node.js and there it supports passing a file name as a file argument. The file will be opened, read and uploaded as a media file.

https://github.com/Automattic/wpcom.js/blob/master/lib/site.media.js#L23

@samouri samouri added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. [Status] Needs e2e Testing (BETA) and removed [Status] In Progress labels Feb 21, 2018
new webpack.DefinePlugin( {
'process.env.NODE_ENV': JSON.stringify( bundleEnv ),
process: { env: { NODE_ENV: JSON.stringify( bundleEnv ) } },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a not that webpack advises against this change

https://webpack.js.org/plugins/define-plugin/#feature-flags

[this way it has changed] will overwrite the process object which can break compatibility with some modules that expect other values on the process object to be defined.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not also that this directly undoes the cleanup from #15691

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I was going to reply that this is intentional but clearly not ideal. It makes sense to only overwrite a specific property as a constant when polyfilling process, but what about when all of process should be undefined because we've disabled the polyfill?

I'm revisiting and trying to make it so that none of our code depends on any non-explicitly included polyfills...which I didn't realize is harder than I thought. node: false doesn't seem to be working like I thought it would!

Things like import path from 'path' and import qs from 'querystring' still work!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've re-cleaned it up. Thank you for pointing this out

@samouri samouri added [Status] In Progress and removed [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. [Status] Needs e2e Testing (BETA) labels Feb 21, 2018
@samouri samouri changed the base branch from remove/stream-browserify to master February 21, 2018 19:56
@samouri samouri force-pushed the update/webpack/node-false branch 2 times, most recently from 2692463 to 47d2ae6 Compare February 21, 2018 20:02
@samouri
Copy link
Contributor Author

samouri commented Feb 21, 2018

in its current form, this pr drops build by 5 kBgz. Probably from changing lru-cache to lru and removing the path polyfill.

Commit: 47d2ae6cdf5a6337b4b7a430dce941969d53d0e3
Author: samouri
At: 2018-02-21T20:02:09Z
Message: remove path deps

Delta:
chunk                                        stat_size           parsed_size           gzip_size
async-load-blocks-reader-full-post               +10 B  (+0.0%)       -644 B  (-1.4%)      +22 B  (+0.2%)
async-load-components-happychat                   +0 B                 -60 B  (-0.2%)      -24 B  (-0.3%)
async-load-components-web-preview-component     +100 B  (+0.0%)      -3736 B  (-1.6%)     -254 B  (-0.5%)
async-load-design                                 +0 B                 -60 B  (-0.0%)      -23 B  (-0.0%)
async-load-design-blocks                      +17060 B  (+0.7%)      +4093 B  (+0.3%)    +1462 B  (+0.5%)
async-load-lib-happychat-connection               +0 B                -567 B  (-0.8%)      -80 B  (-0.4%)
async-load-post-editor-editor-location            +0 B                  -4 B  (-0.1%)       -4 B  (-0.2%)
async-load-post-editor-media-modal               -25 B  (-0.0%)       -122 B  (-0.0%)      -50 B  (-0.1%)
async-load-reader-following-manage                +0 B                 -24 B  (-0.0%)       -7 B  (-0.0%)
build                                         -35225 B  (-1.0%)     -19585 B  (-1.3%)    -4900 B  (-1.3%)
comments                                          +0 B                 -60 B  (-0.0%)      -27 B  (-0.1%)
happychat                                        -12 B  (-0.0%)       -146 B  (-0.1%)      -69 B  (-0.2%)
help                                             -12 B  (-0.0%)        -86 B  (-0.0%)      -42 B  (-0.1%)
manifest                                          +0 B                  +0 B                +0 B
me                                               -12 B  (-0.0%)        -86 B  (-0.1%)      -51 B  (-0.2%)
media                                            -25 B  (-0.0%)       -122 B  (-0.1%)      -39 B  (-0.1%)
notification-settings                            -12 B  (-0.0%)        -86 B  (-0.0%)      -44 B  (-0.1%)
post-editor                                   +17035 B  (+0.7%)      +5180 B  (+0.5%)    +1595 B  (+0.6%)
preview                                          +95 B  (+1.5%)       -210 B  (-4.7%)      -60 B  (-3.4%)
purchases                                        -12 B  (-0.0%)        -86 B  (-0.0%)      -49 B  (-0.0%)
reader                                           +25 B  (+0.0%)      -1127 B  (-0.3%)      +50 B  (+0.1%)
security                                         -12 B  (-0.0%)        -86 B  (-0.0%)      -64 B  (-0.1%)
settings                                          +0 B                  +0 B                +0 B
settings-traffic                               +6473 B  (+0.9%)      +1189 B  (+0.4%)     +560 B  (+0.8%)
settings-writing                                  +0 B                 -59 B  (-0.0%)      -24 B  (-0.0%)
sharing                                           +0 B                  -4 B  (-0.0%)       -4 B  (-0.0%)
stats                                            +10 B  (+0.0%)      -1370 B  (-0.4%)     -156 B  (-0.2%)
themes                                           -25 B  (-0.0%)      -3418 B  (-2.0%)     -681 B  (-1.7%)
tinymce-build                                     +0 B                 -60 B  (-0.0%)      -38 B  (-0.0%)
vendor                                         -5246 B  (-0.2%)      -1884 B  (-0.2%)     -627 B  (-0.3%)
woocommerce                                       +0 B                 -64 B  (-0.0%)      -28 B  (-0.0%)
account                                          -12 B  (-0.0%)        -86 B  (-0.0%)      -49 B  (-0.1%)

@jsnajdr
Copy link
Member

jsnajdr commented Feb 22, 2018

Probably from changing lru-cache to lru

This could be a separate PR. The lru package indeed looks more reasonable for our purposes than lru-cache.

@samouri
Copy link
Contributor Author

samouri commented Feb 22, 2018

separate pr for the lru swap: #22714

@samouri
Copy link
Contributor Author

samouri commented Feb 23, 2018

update
I'm going to switch gears a bit with this PR now that I'm armed with more knowledge. My goal with this PR as a whole was to setup preventative checks so that we don't import nodejs polyfills without it being a deliberate decision. Along the way I found out that even though node: false in the webpack config will stop webpack from polyfilling, it won't break the build if we have modules importing from things like path or querystring. That is because they can be pulled in as transitive dependencies and webpack does not forbid direct dependence on transitive deps.

The good news is that there are eslint plugins that can accomplish exactly what we want. By setting node: false, and adding the eslint rules import/no-nodejs-libs and import/no-extraneous-dependencies, we can ensure that:

  1. We don't directly depend on transitive dependencies
  2. We don't use any nodejs libs

@samouri samouri force-pushed the update/webpack/node-false branch 4 times, most recently from 3ce867c to 52aeba2 Compare February 23, 2018 17:17
@samouri
Copy link
Contributor Author

samouri commented Mar 1, 2018

@blowery w.r.t global:
I'm going to try to get rid of all occurrences of global since I don't think they belong on the web.

  1. upgrade uuid to 3.2.1 which gets rid of global: using rng-browser.js: global is not defined (3.1.0) uuidjs/uuid#252
  2. remove global from progress-event which is depended on by wpcom-rest-proxy: Browser compatibility without shimming global webmodules/progress-event#1
  3. remove global from wp-calypso: 11 non-test files: Remove all references to global in client code #22899
  4. ... there are more libs that also reference global. Namely any lib older than a couple years that was meant for CommonJS. Like twemoji: global is undefined twitter/twemoji#218, etc

edit: maybe just do a DefinePlugin for global --> window?

@samouri samouri force-pushed the update/webpack/node-false branch 2 times, most recently from d4036f9 to 41e909b Compare March 1, 2018 19:44
@samouri
Copy link
Contributor Author

samouri commented Mar 1, 2018

final update:
There are some really beneficial things to land in this PR sooner rather than later.
Because of that I want to defer some of the less impactful ideas in favor of getting the main thrust of this PR.

included

  • node: false to stop webpack from polyfilling
  • eslint rules to stop us from depending on transitive dependencies and from depending on banned nodejs libs
  • global --> window DefinePlugin. Also made many window calls safer.
  • removal of fs mock.
  • process.nextTick --> setTimeout( fn, 0 )

deferred:

  • removal of events
  • removal of path
  • removal of url

ICFY: http://iscalypsofastyet.com/branch?branch=update/webpack/node-false

chunk                                        stat_size           parsed_size           gzip_size
async-load-blocks-reader-full-post               +10 B  (+0.0%)       -651 B  (-1.4%)      +34 B  (+0.3%)
async-load-components-happychat                   +0 B                 -64 B  (-0.2%)      -29 B  (-0.3%)
async-load-components-web-preview-component     +154 B  (+0.0%)      -3717 B  (-1.7%)     -311 B  (-0.6%)
async-load-design                                 +0 B                 -64 B  (-0.0%)      -26 B  (-0.0%)
async-load-design-blocks                         +63 B  (+0.0%)      -1259 B  (-0.1%)     -114 B  (-0.0%)
async-load-layout-masterbar-drafts-popover        -1 B  (-0.0%)         +0 B                +0 B
async-load-lib-happychat-connection               +0 B                -708 B  (-1.0%)      -92 B  (-0.5%)
async-load-post-editor-editor-location            +0 B                  -4 B  (-0.1%)       -4 B  (-0.2%)
async-load-post-editor-media-modal                +0 B                 -97 B  (-0.0%)      -37 B  (-0.1%)
async-load-reader-following-manage                +0 B                 -44 B  (-0.0%)      -10 B  (-0.0%)
build                                          -9155 B  (-0.3%)      -6428 B  (-0.4%)    -1239 B  (-0.3%)
checkout                                          +0 B                  +1 B  (+0.0%)       +2 B  (+0.0%)
comments                                          +0 B                 -64 B  (-0.0%)      -30 B  (-0.1%)
devdocs                                           -1 B  (-0.0%)         +0 B                +0 B
happychat                                         +0 B                 -64 B  (-0.0%)      -28 B  (-0.1%)
manifest                                          +0 B                  +0 B                +0 B
media                                             +0 B                 -97 B  (-0.0%)      -36 B  (-0.1%)
post-editor                                       -3 B  (-0.0%)       -164 B  (-0.0%)      -77 B  (-0.0%)
posts-custom                                      -1 B  (-0.0%)         +0 B                +0 B
posts-pages                                       -1 B  (-0.0%)         +0 B                +0 B
preview                                         +159 B  (+2.5%)       -198 B  (-4.5%)      -55 B  (-3.1%)
reader                                           +63 B  (+0.0%)      -1118 B  (-0.3%)      +60 B  (+0.1%)
settings-traffic                               +6473 B  (+0.9%)      +1185 B  (+0.4%)     +566 B  (+0.8%)
settings-writing                                  +0 B                 -97 B  (-0.0%)      -36 B  (-0.1%)
sharing                                           +0 B                  -4 B  (-0.0%)       -4 B  (-0.0%)
signup                                            +0 B                  +1 B  (+0.0%)       +1 B  (+0.0%)
stats                                             +4 B  (+0.0%)      -1382 B  (-0.4%)     -181 B  (-0.2%)
themes                                           -16 B  (-0.0%)      -3421 B  (-1.9%)     -688 B  (-1.7%)
tinymce-build                                     +0 B                 -60 B  (-0.0%)      -38 B  (-0.0%)
vendor                                         -5927 B  (-0.3%)      -1983 B  (-0.2%)     -651 B  (-0.3%)
woocommerce                                       +0 B                 -68 B  (-0.0%)      -32 B  (-0.0%)

to test

  1. smoke test clicking around the app. no unexpected errors should show up in the dev console.
  2. try including a transitive dependency in a file. you should see an eslint error
  3. try including nodejs libs like http. You should both get a compilation error and an eslint error

@samouri samouri added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. and removed [Status] In Progress labels Mar 1, 2018
@samouri
Copy link
Contributor Author

samouri commented Mar 1, 2018

because calypso.live is breaking on this branch name, I've duped this branch on #22925 just for e2e and canary tests

Copy link
Member

@jsnajdr jsnajdr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very impressive cleanup indeed! Thanks for working on this.

@@ -101,7 +104,8 @@ export default class Step extends Component {
this.quitIfInvalidRoute( nextProps, nextContext );
this.skipIfInvalidContext( nextProps, nextContext );
this.scrollContainer.removeEventListener( 'scroll', this.onScrollOrResize );
this.scrollContainer = query( nextProps.scrollContainer )[ 0 ] || global.window;
this.scrollContainer =
query( nextProps.scrollContainer )[ 0 ] || ( typeof window !== 'undefined' && window );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the typeof window check necessary? If window is undefined, then the app will crash on the very next line anyway (addEventListener).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it may not be necessary. I know sometimes it is to prevent tests from failing.
seems like the typeof check isn't covering enough in this case.

Since tests are all passing, I can remove this check and someone can add it back in if the future if we ever need it

@@ -3,7 +3,7 @@
/**
* External dependencies
*/
import assert from 'assert';
import assert from 'assert'; // eslint-disable-line import/no-nodejs-modules
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be awesome and quite simple to migrate to import { assert } from 'chai' here. But that's for another PR. A codemod would help a lot with the migration for sure.

Copy link
Contributor Author

@samouri samouri Mar 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a small multi-file find/replace would probably work as well.
I'll leave that for a small follow-up pr

// file. Webpack uses the same module on the client side, too, which
// makes for a nice consistency.
import { EventEmitter } from 'events/';
import { EventEmitter } from 'events';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you verify somehow that the domain property on EventEmitter-decorated objects is no longer a threat? The Site class mentioned in the comment is gone, but there might be other usages that are still live.

Copy link
Contributor Author

@samouri samouri Mar 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be frank: I don't understand the comment or how it could make sense.
webpack never bundled the actual "node module". It always used a web polyfill. In this specific case it was Gozala/events which is published to npm as events. Therefore I don't think the / would change anything...

Also, if you look at the source of events, you won't find domain anywhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed on slack. I think the comment was largely historical based on where this file used to live in the pre-oss days

@@ -48,11 +48,15 @@ class PreviewMain extends React.Component {
debouncedUpdateLayout = debounce( this.updateLayout, 50 );

componentDidMount() {
global.window && global.window.addEventListener( 'resize', this.debouncedUpdateLayout );
if ( typeof window !== 'undefined' ) {
window && window.addEventListener( 'resize', this.debouncedUpdateLayout );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking the window truthiness for a second time is not necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove the second check

@samouri samouri merged commit a560773 into master Mar 2, 2018
@samouri samouri deleted the update/webpack/node-false branch March 2, 2018 18:10
@matticbot matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Mar 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants