core: add desktop score curves for TBT and LCP #10756

brendankenny · 2020-05-12T19:06:04Z

After lining up http archive mobile and desktop distributions with the Lighthouse distribution, doing some simple regressions, then crossing my eyes and transcribing cosmic ray damage to my retinas as it occurred, we have some new desktop curves for TBT and LCP.

They seem ok.

part of #9774

connorjclark · 2020-05-12T19:31:25Z

lighthouse-core/audits/metrics/largest-contentful-paint.js

-      p10: 2500,
-      median: 4000,
+      mobile: {
+        // 25th and 13th percentiles HTTPArchive -> median and p10 points.


why is this 13?

is this the x percentile of all web traffic or just desktop? (same question for mobile control points). comment doesn't make it clear, but the link to bigquery specifies mobile so I think it's filtering device type?

why is this 13?

#10715 changed the second control point from the PODR to p10, but in order to not change the curves, the p10 point ended up at a somewhat arbitrary percentile for each audit.

(this PR isn't changing any of the mobile numbers, though (?w=1 makes this a little clearer))

is this the x percentile of all web traffic or just desktop? (same question for mobile control points). comment doesn't make it clear, but the link to bigquery specifies mobile so I think it's filtering device type?

Right, the mobile and desktop data come from the percentiles in the respective http archive tables (*_mobile and *_desktop) mentioned in each comment.

~~I think this was decided to leave the LCP points as they were at some point prior and this is just a noop whitespace change.~~

Commented at the same time as @brendankenny but he explained better :)

patrickhulce · 2020-05-12T19:36:43Z

lighthouse-core/audits/metrics/largest-contentful-paint.js

+        },
+      },
+      desktop: {
+        // SELECT QUANTILES(lcp, 10), QUANTILES(lcp, 30) FROM [httparchive.pages.2020_04_01_desktop]


did you update the query here @brendankenny? I'm a little confused.

AFAICT it's supposed to return every ~11th quantile and then every ~3.3rd quantile but which ones were used? also I get validation errors that lcp isn't a property in the pages table

shouldn't we use 25th and 5th if we can and stick to the QUANTILES(lcp, 21) query?

would also be nice to standard sql in our new queries moving forward and use APPROX_QUANTILES but no biggie that's a separate topic :)

AFAICT it's supposed to return every ~11th quantile and then every ~3.3rd quantile but which ones were used?

the HTTP Archive WPT desktop/mobile distributions are far enough away from the Lighthouse distributions that we can't really base it just on the desktop numbers so simply I don't think. As an example, the pages_mobile numbers would give {p10: 4648, median: 5978} for the mobile scoring control points here (vs our 2500/4000). That's why I mentioned comparing those curves to the Lighthouse curve, then I tried to pick analogous points to our mobile control points based on the pages_desktop curve.

shouldn't we use 25th and 5th if we can and stick to the QUANTILES(lcp, 21) query?

But these are pretty close to my results ({p10: 1193, median: 2419}) so let's ignore what I thought was useful and let's keep it somewhat standardized :)

would also be nice to standard sql in our new queries moving forward and use APPROX_QUANTILES but no biggie that's a separate topic :)

ha, I forgot that's actually the legacy SQL function and thought it was just a shorthand for APPROX_QUANTILES and an offset :) I can update to that.

also I get validation errors that lcp isn't a property in the pages table

yeah, it's not in any of the summaries, this is going into the json payload and getting "$['_chromeUserTiming.LargestContentfulPaint']". I can include that as well.

this is going into the json payload and getting "$['_chromeUserTiming.LargestContentfulPaint']". I can include that as well.

Gotcha, ya working queries would be 👌 (must imagine Paul's signature noise while reading that)

patrickhulce · 2020-05-12T19:45:05Z

lighthouse-core/audits/metrics/total-blocking-time.js

+        },
+      },
+      desktop: {
+        // SELECT QUANTILES(tbt, 40), QUANTILES(tbt, 60) FROM [httparchive.pages.2020_04_01_desktop]


ah wait a second, are you trying to say we should use 40th and 60th percentiles? I don't see tbt as a property either though :/

ah wait a second, are you trying to say we should use 40th and 60th percentiles? I don't see tbt as a property either though :/

this is $._TotalBlockingTime, but our mobile control points are way off from our standard method of choosing control points (see Deep's original comment above about the mobile control points).

Our normal process would give {p10: 0, median: 83} for the desktop curve. We could do that but that seems way too harsh, so I chose points on the desktop curve adjusted somewhat similarly to the existing mobile ones.

Happy to change if you feel differently, though.

I wasn't asking because of the specific quantiles chosen :)

I forgot that's actually the legacy SQL function and thought it was just a shorthand for APPROX_QUANTILES and an offset

Is the single sentence that explains 70% of my confusion. You were trying to use quantiles to return a single offset so 40th and 60th was the goal here rather than "give me 40 quantiles and then 60 quantiles". It all makes sense now!

I wasn't asking because of the specific quantiles chosen :)

ha, ok, I see now too :) Yeah, I'll update to make it actually clear

patrickhulce · 2020-05-12T19:46:18Z

lighthouse-core/test/audits/metrics/largest-contentful-paint-test.js

+    const contextMobile = generateContext({throttlingMethod: 'simulate'});
+
+    const outputMobile = await LCPAudit.audit(artifactsMobile, contextMobile);
+    expect(outputMobile.numericValue).toBeCloseTo(2758.2, 1);


if we're using lantern for this number we should use snapshots, if you just want to test the scoring stuff, maybe use throttlingMethod: 'provided'?

patrickhulce · 2020-05-12T19:46:22Z

lighthouse-core/test/audits/metrics/largest-contentful-paint-test.js

+    const contextDesktop = generateContext({throttlingMethod: 'simulate'});
+
+    const outputDesktop = await LCPAudit.audit(artifactsDesktop, contextDesktop);
+    expect(outputDesktop.numericValue).toBeCloseTo(2758.2, 1);


patrickhulce · 2020-05-12T19:46:32Z

lighthouse-core/test/audits/metrics/total-blocking-time-test.js

+
+  it('adjusts scoring based on form factor', async () => {
+    const artifactsMobile = generateArtifacts({trace, devtoolsLog, TestedAsMobileDevice: true});
+    const contextMobile = generateContext({throttlingMethod: 'simulate'});


same here

yeah, lcp was just c/p from here and is fine to switch, but for TBT I was just trying to slow things down enough to get a scoring difference (with provided both are 1). I'll find some other trace to test.

patrickhulce · 2020-05-12T19:48:23Z

lighthouse-core/audits/metrics/largest-contentful-paint.js

-      p10: 2500,
-      median: 4000,
+      mobile: {
+        // 25th and 13th percentiles HTTPArchive -> median and p10 points.


~~I think this was decided to leave the LCP points as they were at some point prior and this is just a noop whitespace change.~~

Commented at the same time as @brendankenny but he explained better :)

patrickhulce

queries look great, much better than my sad QUANTILES(renderStart, 21) too :)

LGTM

core: add desktop score curves for TBT and LCP

d00e0bc

brendankenny requested a review from a team as a code owner May 12, 2020 19:06

brendankenny requested review from Beytoven and removed request for a team May 12, 2020 19:06

googlebot added the cla: yes label May 12, 2020

devtools-bot assigned Beytoven May 12, 2020

devtools-bot added the waiting4reviewer label May 12, 2020

brendankenny mentioned this pull request May 12, 2020

[meta] Lighthouse 6.0 Burndown #9774

Closed

54 tasks

connorjclark reviewed May 12, 2020

View reviewed changes

patrickhulce suggested changes May 12, 2020

View reviewed changes

review feedback

7616223

vercel bot deployed to Preview May 12, 2020 21:43 View deployment

paulirish added the 6.0 label May 12, 2020

patrickhulce approved these changes May 13, 2020

View reviewed changes

brendankenny added land-when-ci-is-green and removed waiting4reviewer labels May 13, 2020

devtools-bot merged commit 95898ad into master May 13, 2020

devtools-bot deleted the desktopcurves branch May 13, 2020 17:30

devtools-bot removed the land-when-ci-is-green label May 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: add desktop score curves for TBT and LCP #10756

core: add desktop score curves for TBT and LCP #10756

brendankenny commented May 12, 2020

connorjclark May 12, 2020

brendankenny May 12, 2020

patrickhulce May 12, 2020 •

edited

Loading

patrickhulce May 12, 2020

brendankenny May 12, 2020

patrickhulce May 12, 2020

patrickhulce May 12, 2020

brendankenny May 12, 2020

patrickhulce May 12, 2020

brendankenny May 12, 2020

patrickhulce May 12, 2020

patrickhulce May 12, 2020

patrickhulce May 12, 2020

brendankenny May 12, 2020

patrickhulce May 12, 2020 •

edited

Loading

patrickhulce left a comment

core: add desktop score curves for TBT and LCP #10756

core: add desktop score curves for TBT and LCP #10756

Conversation

brendankenny commented May 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce May 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce May 12, 2020 • edited Loading

Choose a reason for hiding this comment

patrickhulce left a comment

Choose a reason for hiding this comment

patrickhulce May 12, 2020 •

edited

Loading

patrickhulce May 12, 2020 •

edited

Loading