perf: request block stylesheets and images on redirect pass #2168

patrickhulce · 2017-05-05T23:58:41Z

shaves off ~10s on certain sites like theverge.com

paulirish

nice cleanup on the test. :)
oh and fixing the fact that addBlockedUrls hasn't worked for a few months now. :)

we're not explicitly clearing our blocked urls at the end of a pass, but instead relying on a subsequent pass doing Network.setBlockedUrls({ urls: [] }) to clear things out. Can you add a comment on the blockUrlPatterns method that mentions this trick?
(it seems fine but it gave me a doubletake)

paulirish · 2017-05-06T17:15:32Z

lighthouse-core/config/default.js

@@ -27,6 +27,18 @@ module.exports = {
  {
    "passName": "redirectPass",
    "useThrottling": false,
+    // Speed up the redirect pass by blocking stylesheets, fonts, and images
+    "blockedUrlPatterns": [
+      "*.css",


formatting nit: this is pretty low signal info, compared to the rest of config.
can these all share a line?

paulirish · 2017-05-06T17:27:54Z

lighthouse-core/gather/gather-runner.js

@@ -45,7 +45,8 @@ let GathererResults; // eslint-disable-line no-unused-vars
 * 2. For each pass in the config:
 *   A. GatherRunner.beforePass()
 *     i. navigate to about:blank
- *     ii. all gatherers' beforePass()
+ *     ii. Enable network request blocking for specified patterns


set patterns for network request blocking

and you can remove "vii. blockUrlPatterns" from above.

paulirish · 2017-05-06T17:29:22Z

lighthouse-core/gather/driver.js

-  blockUrlPatterns(urlPatterns) {
-    const promiseArr = urlPatterns.map(url => this.sendCommand('Network.addBlockedURL', {url}));
-    return Promise.all(promiseArr);
+  blockUrlPatterns(urls) {


jsdoc?

{!Array<string>} yeah?

patrickhulce · 2017-05-08T16:15:41Z

Can you add a comment on the blockUrlPatterns method that mentions this trick?

Yeah good call done, blocking will cease when we disconnect, right?

paulirish

one this is done we'll be good

paulirish · 2017-05-08T17:59:27Z

lighthouse-core/gather/driver.js

+   * @return {!Promise}
+   */
+  blockUrlPatterns(urls) {
+    return this.sendCommand('Network.setBlockedURLs', {urls});


unfortunately this command was renamed/added recently..

commit: https://chromium.googlesource.com/chromium/src/+/01c043f5f744262981becd3d45a79981ad3f6b67

m58 (base rev 454471 ) which shipped to stable 2 weeks ago doesn't have this method. It has the add/remove pair instead.

IMO this is optional, and our other use of requestblocking is unused so i think we can ignore the failure. let's catch and exclude errors?

doublecheck this works with a stable m58 build

what's going to be our policy going forward about handling stable chrome and protocol changes? it seems like it's mostly taken a backseat and we emphasize canary, should we add a stable config to travis if we want to support it (#1480)?

confirmed working now (with an error log) in 58

brendankenny

nice clean perf improvement. LGTM with comment location shuffling

brendankenny · 2017-05-08T18:23:38Z

lighthouse-core/gather/driver.js

-    return Promise.all(promiseArr);
+  /**
+   * This method sets the URL patterns to be blocked and is called at the beginning of each pass.
+   * No "clearing" is done at the end of the pass since the next pass will set to [] if necessary.


since some future gatherers or other consumers of lighthouse (e.g. performanceExperiment) may want to use this method, probably makes more sense to move the bulk of this comment into beforePass where it's used like this

brendankenny · 2017-05-08T18:23:59Z

lighthouse-core/gather/driver.js

+  /**
+   * This method sets the URL patterns to be blocked and is called at the beginning of each pass.
+   * No "clearing" is done at the end of the pass since the next pass will set to [] if necessary.
+   * @param {!Array<string>} urls


maybe worth adding param doc similar to what the protocol docs has? ("URL patterns to block. Wildcards ('*') are allowed.")

brendankenny · 2017-05-08T18:25:50Z

lighthouse-core/gather/gather-runner.js

@@ -171,9 +170,12 @@ class GatherRunner {
   * @return {!Promise}
   */
  static beforePass(options, gathererResults) {
+    const blockedUrls = (options.config.blockedUrlPatterns || [])
+      .concat(options.flags.blockedUrlPatterns || []);


how do unions of url patterns work? Last one wins? (just curious)

aside: we really need to get the overloaded naming of options under control here :)

how do unions of url patterns work? Last one wins?

oh, is ! not supported? I guess in that case it doesn't matter

! isnt supported, correct.

brendankenny · 2017-05-08T18:29:12Z

lighthouse-core/config/default.js

@@ -27,6 +27,8 @@ module.exports = {
  {
    "passName": "redirectPass",
    "useThrottling": false,
+    // Speed up the redirect pass by blocking stylesheets, fonts, and images


love having this in the default config as an example for custom configs

brendankenny · 2017-05-08T20:16:28Z

lighthouse-core/config/default.js

@@ -27,6 +27,8 @@ module.exports = {
  {
    "passName": "redirectPass",
    "useThrottling": false,
+    // Speed up the redirect pass by blocking stylesheets, fonts, and images
+    "blockedUrlPatterns": ["*.css", "*.jpg", "*.jpeg", "*.png", "*.gif", "*.svg", "*.ttf", "*.woff", "*.woff2"],


not sure about this one, but does it make sense to have these apply across beforePass, pass, and afterPass? Maybe it would make more sense/be less surprising to have them blocked only during pass, analogous to recordTrace, useThrottling, etc cc/ @paulirish

yeah few random thoughts here

we don't have any gatherers that load new content that wasn't already requested by the page

we should probably discourage that behavior as something outside of Lighthouse realm anyway, and suggest requesting on the node side if a third party integration wants to do that rather than an evaluateAsync(fetch('my-url'))

since afterPass doesn't freeze the page or anything and we're not guaranteed to be done at the 25s timeout it felt weird to unblock these urls while the page could feasibly still be requesting or retrying them

we don't have any gatherers that load new content that wasn't already requested by the page

that's actually why I brought it up, so it wouldn't surprise us when someone does write a gatherer that tries it :)

we should probably discourage that behavior as something outside of Lighthouse realm anyway, and suggest requesting on the node side if a third party integration wants to do that rather than an evaluateAsync(fetch('my-url'))

but that's definitely something we rely on when we need to get around authentication or whatever. As an easy workaround to some of those issues, fetching in the context of the page can't be beat.

since afterPass doesn't freeze the page or anything and we're not guaranteed to be done at the 25s timeout it felt weird to unblock these urls while the page could feasibly still be requesting or retrying them

yeah, good point, and this is what really separates it from useThrottling. While matching their behavior makes some sense from a user-surprise perspective, disabling useThrottling in afterPass means the run goes faster due to potential work going faster, while disabling url blocking potentially means the run going slower due to (potentially) making work.

My real hesitation is magic while having no data about actual downsides. This is why we need users making custom gatherers :)

In the meantime, I suspect we'll have to revisit scheduling of all of this for #1826 anyways, so I'm good with this.

perf: request block stylesheets and images on redirect pass

ec74058

patrickhulce added the OYB label May 5, 2017

paulirish approved these changes May 6, 2017

View reviewed changes

paulirish mentioned this pull request May 6, 2017

[meta] I/O The Final Countdown #2161

Closed

40 tasks

nits and comments

c2f92c3

paulirish requested changes May 8, 2017

View reviewed changes

add catch for m58

c93b2fb

brendankenny approved these changes May 8, 2017

View reviewed changes

brendankenny reviewed May 8, 2017

View reviewed changes

move comments

091f82b

brendankenny mentioned this pull request May 8, 2017

perf(gather-runner): Clear cache selectively per-pass #2156

Merged

brendankenny assigned paulirish May 8, 2017

paulirish approved these changes May 9, 2017

View reviewed changes

paulirish merged commit 4c515cf into master May 9, 2017

paulirish deleted the faster_redirect_pass branch May 9, 2017 19:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: request block stylesheets and images on redirect pass #2168

perf: request block stylesheets and images on redirect pass #2168

patrickhulce commented May 5, 2017

paulirish left a comment

paulirish May 6, 2017

patrickhulce May 8, 2017

paulirish May 6, 2017

patrickhulce May 8, 2017

paulirish May 6, 2017

patrickhulce May 8, 2017

patrickhulce commented May 8, 2017

paulirish left a comment

paulirish May 8, 2017

patrickhulce May 8, 2017

brendankenny left a comment

brendankenny May 8, 2017

patrickhulce May 8, 2017

brendankenny May 8, 2017

patrickhulce May 8, 2017

brendankenny May 8, 2017

brendankenny May 8, 2017

brendankenny May 8, 2017

paulirish May 8, 2017

brendankenny May 8, 2017

brendankenny May 8, 2017

patrickhulce May 8, 2017

brendankenny May 8, 2017

perf: request block stylesheets and images on redirect pass #2168

perf: request block stylesheets and images on redirect pass #2168

Conversation

patrickhulce commented May 5, 2017

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented May 8, 2017

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment