Console errors from Chrome extensions fail the errors-in-console audit

[adam-buckley]

Running Lighthouse bundled with Chrome 68.0.3440.84 (3.0.0-beta.0) on Windows 10.

My apologies if this issue exists, I did a few searches for various keywords and nothing came up that was relevant.

Provide the steps to reproduce

1. Run LH on any site with an extension installed that is generating JS errors.

What is the current behavior?

Any site that I profile for Best Practices gets penalised with the category "Browser errors were logged to the console" but the error comes from an extension that I have installed.

My error specifically relates to the Last Pass extension throwing a JS TypeError, but this issue should apply to any extension that has a JS error.

What is the expected behavior?

Lighthouse in Chrome should ignore javascript errors that originate from extensions and/or code outside of the site owners control.

It's misleading because I could run Lighthouse on a fresh Chrome install with no extensions and get a 100 score in Best Practices, but on my browser as it is is instead giving a score of 94.

Environment Information

• Affected Channels: DevTools
• Lighthouse version: 3.0.0-beta.0
• Node.js version: N/A?
• Operating System: Windows 10 Version 10.0.17134 Build 17134

[justinribeiro]

This comes up from time to time; the recommendation is to run LH in a clean Chrome profile or incognito when diagnosing performance issues. With the recently landed #5666, in situations where extensions are causing a performance impact on boot time, this will be surfaced in the report (the aforementioned ticket has further discuss and a screenshot of this behavior).

[brendankenny]

Yeah, we should filter out the console errors from any extensions and not count them against the audit score (though maybe they should still be listed).

I'm actually a little surprised we didn't give errors-in-console a weight of 0 anyways, given how tenuously you could call it a "best practice" :)

[adam-buckley]

Thanks @justinribeiro and @brendankenny, I'll run it incognito for now. I would want to see console errors originating from the site I'm auditing in the report, I don't mind if it penalises either for errors that myself/a developer is responsible for. But I think errors from extensions are entirely situational and could be ignored completely.

[wardpeet]

@brendankenny should we set errors-in-console to 0?

wdyt @patrickhulce @paulirish ?

[paulirish]

I think it's reasonable to keep punishing for having errors. Personally that's probably how i would have learned about the exception on paulirish.com.

Agree with excluding extensions from it, though.

[patrickhulce]

If we're filtering extensions I think we should make a blanket decision to disable extensions across the board rather than ad-hoc some audits ignore them others don't.

I will certainly miss the ability to analyze the impact of extensions on the page though. Ideally I'd like to see this as an option that's on by default :)

[brendankenny]

Maybe we should keep the audit the same and add the same warning that bootup-time got. The only issue is deduping the warnings so you don't get like 5 warnings that you have extensions running

[adam-buckley]

@patrickhulce the impact of extensions on the page is entirely local to your machine and I don't think it's a reliable source of effect to the overall performance of the page.

I would love to see extensions disabled by default, my way around this for now is incognito so I know the audits are consistent with regards to extensions.

I appreciate everyone giving this some time.

[wardpeet]

@adam-buckley I agree we shouldn't show the errors of an extension in the audit but we should still show a warning that you should run without extensions as it might only be local but it would skew perf results and give different results on Webpagetest or Cli.

[patrickhulce]

the impact of extensions on the page is entirely local to your machine and I don't think it's a reliable source of effect to the overall performance of the page.

From the perspective of a page owner, sure! But there are extension developers that should care about the impact they're having on their users too :)

I still wholeheartedly agree with making the page owner's perspective and concerns the default experience since that's the overwhelmingly common case. I'm just saying that as a general purpose performance analysis tool that reports on what it sees, it'd be unfortunate to make it impossible for set of developers to use Lighthouse when a workaround exists today.

[adam-buckley]

@wardpeet is there a way to disable extensions when Lighthouse is run? As @patrickhulce says page owners running Lighthouse are the overwhelming majority, it would great to have the option to disable extensions in the Lighthouse setup.

it'd be unfortunate to make it impossible for set of developers to use Lighthouse when a workaround exists today

I don't want to see extensions completely removed from the equation, that doesn't make sense. But being able to turn extensions on or off in reports would work for both page and extension developers.

[paulirish]

We'd like to handle the impact of extensions consistently across this audit and the perf ones. We plan to publish docs on why we wont auto-filter ext/3P results from the report, but will allow you to visually hide them (without recomputing score). One can always run in incognito to remove impact from extensions.

closing into #4516