When lighthouse cannot reach site, lighthouse-ci reports all tests passing

[glassdimly]

What happens
Here's what lighthouse-ci generates:

Here's what lighthouse cli generates from the same site.

lighthouse http://local.mysite.org
  ...
Runtime error encountered: Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 503)

What should happen
If lighthouse returns a 503, lighthouse-ci should report a fail, or fail all tests.

[andreasonny83]

First of all, thanks for your help @glassdimly

To be honest, that doesn't really sound like a bug to me.
The problem is that, when providing an invalid URL to Chrome, the page is still rendering something depending on your Internet provider. You cannot see an error page directly in Chrome as it defaults to some content instead.

Also, checking the existence of the website could be generally done before running performance tests, via E2E tests or integrations.

Closing this issue for now. Please, feel free to reopen if you don't agree or you have more questions.