Write Lighthouse reports to a new *_lighthouse table

[rviscomi]

To summarize the latest changes in this PR:

• LH reports are no longer written to the *_pages table
• reportCategories..audits are omitted
• screenshot-thumbnails items (base64 encoded images) are omitted
• report is NULL rather than the string "null" when applicable

These are efforts to minimize the row size of the LH report so that we can fit more reports. Here are the number of skipped rows per optimization:

• none: ~12k
• omit audits: ~4.8k
• ^ + omit images: ~4.6k
• ^ + separate table: ~4.6k

[igrigorik]

@rviscomi can you share an example of what these duplicates are?

[rviscomi]

@igrigorik here's the same color-contrast audit in two places

[igrigorik]

Hmm, @ebidel why is this data duplicated in LH reports? I imagine optimizing report size is not high on the agenda for LH, but duplication still feels a bit odd.

[ebidel]

Known issue: GoogleChrome/lighthouse#1999

More info: when we render a report in LH, it's easier that the related audits to the category are nested under reportCategories.audits. But that's something we just need to fix. Instead, reference the audits by their ids.

[igrigorik]

Gotcha, thanks Eric. Would be great to get this fixed upstream.. With more and more tools integrating LH reporting, I imagine this will become a recurring pain point.

[ebidel]

I'm going to start the work upstream to remove the dupes. No promises on timeline though :)

[rviscomi]

Updated to use a separate HAR table. Doesn't seem to have an effect on the number of skipped rows, surprisingly. Also, ~50k pages have NULL reports. The 6/15 table without the changes in this PR has only ~33k "null" reports.

One positive side-effect of this change is that the processing takes the normal amount of 45 minutes, rather than 2.5 hours.

[igrigorik]

Faster import time is a nice win. Looks like we'll need to find a way to prune the data a bit more? =/

[ebidel]

FWIW, I see only 18 tables with non-null reports for 6/20:

SELECT
  count(*)
FROM
  httparchive.har.2017_06_20_android_lighthouse
WHERE
  report != 'null'

[rviscomi]

Sorry you can ignore the 6/20 tables. Those are temporary and just for testing.

[rviscomi]

Here's a list of URLs and ranks for sites omitted from the LH table, presumably because they were too large and skipped: query (results)

Lowest rank is 250 so it does affect some of the more popular sites (eg slate, wsj, digg, theknot, etc).

Using this data, I can analyze these specific HARs for ways to minimize bytes.

[rviscomi]

@igrigorik does this LGTY? It'd be good to standardize the schema for a new LH table for the 7/15 crawl ending soon. I can follow up with optimizations to retain as many reports as possible.

[igrigorik]

Probably a non-event for the most part.. but should probably account for URL size? Other then that, lgtm.

[igrigorik]

Should you account for URL size as well? We do this for body table.

[rviscomi]

Done

[igrigorik]

👍

[rviscomi]

Updated production dataflow worker

[ebidel]

Rick, when do you expect the first dumps to show up?

…

On Fri, Jul 28, 2017, 11:59 AM Rick Viscomi ***@***.***> wrote: Updated production dataflow worker — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOigNZGOfv8NIacsXKGYiay59uxbgAKks5sSi-TgaJpZM4OIhfZ> .

[rviscomi]

@ebidel The 6/1, 6/15, and 7/1 crawls have been beta testing this schema so it's already in prod. The 7/15 crawl looks just about done so it should also be processed in the next few days.