Chrome Canary trace format has changed

[patrickhulce]

Provide the steps to reproduce

1. Run LH CLI on chrome://version using Chrome Canary (if Canary is installed, CLI uses it by default)

Also intermittently on example.com, cnn.com, any URL

What is the current behavior?

All trace-dependent audits fail NO TRACING...

What is the expected behavior?

LH succeeds (though even after fixing this chrome://version has two other problems that need to be fixed, but that's a story for another bug

Environment Information

• Affected Channels: CLI (but probably all)
• Lighthouse version: master (v4.1.0+), but probably all
• Node.js version: v10.12.0
• Operating System: linux

[patrickhulce]

I've uncovered two common scenarios so far.

1. All events we need are in the trace, but the pid of TracingStartedInBrowser event and the CrRendererMain events do not match. This seems to be because we start tracing on about:blank and Chrome now does not always reuse the same process when navigating to a URL. I am not able to determine any pattern about when this happens/does not happen.
2. TracingStartedInBrowser event is just missing entirely. This definitely seems to be a Chrome bug. We obviously started tracing because I'm staring at a 16MB trace file, but the bookkeeping event seems to have vanished.

Attached a few example traces (GH won't let you upload JSON for some dumb reason, so the 3 traces are in this tarball).

traces.tar.gz

[patrickhulce]

I have a workaround for 1, but for 2 I think we'll need some Chrome-side investigation.

[patrickhulce]

After starting 2/2 on example.com and 1/1 on cnn.com, I'm now at 2/10 and 1/5. Not nearly as alarming, but chrome://version still fails 100%.

[patrickhulce]

I threw up my best attempt at bridging this gap in #7122, but I think it really requires someone close to tracing (alph maybe?) to weigh in on what we should do as trace consumers when this happens.

Unassigning myself for now.

[patrickhulce]

This hasn't seemed like it affects anything other than chrome://version so I'm lowering the priority here.

[patrickhulce]

While this doesn't seem to have the same initial affects on Mac/Windows, it appears to have heavily hit linux now that m74 has made it to stable.

Bumping priority back up :)

[connorjclark]

So how can breaking trace changes happen, if Chromium has a test that runs LH (the devtools Audits panel has a few)?

[patrickhulce]

@hoten

// screenshots in content shell are flaky and NO_NAVSTART occurs on bots way more frequently than local
// ignore the results of the trace-dependent audits, just make sure they ran

https://cs.chromium.org/chromium/src/third_party/blink/web_tests/http/tests/devtools/audits2/audits2-successful-run.js?l=6-7

:)