I want to scan a whole site, but only get ~200 results

[mgifford]

Details

I've got a few big sites I'd like to scan, but it keeps stopping about 200 or so pages in.

Is there a way to override that? It's still a good measure, but would be useful to be able to scan the whole site if needed.

Maybe I'm just missing something in the config.

[tuminzee]

can you try to explain this with example? I am not able to understand
unlighthouse is made so that we can generate reports on all routes

[mgifford]

So I run this:

npx unlighthouse --site https://example.com/eng/index.html

All churns on as you'd expect until I get:

✔ Completed runLighthouseTask for /eng/declaration/wwl-cna/c15/index.html. [Score: 0.94 Samples: 1 100% complete]
✔ Unlighthouse has finished scanning https://example.com/eng/index.html/: 200 routes in 431s.

That's for basically any site I craw.

So I get 200 or so scans but the site is much bigger.

[harlan-zw]

There is config for the maximum number of routes to scan scanner.routeRules that is set to 200 by default.

This was implemented as the stability of the worker and the UI starts degrading around here and it's quite easy for a site scan to end up queueing thousands of routes.

I've pushed up a warning that will be triggered when you hit the limit to give better visibility, it will be available in v0.6.0 which will be released soon.

You can read more about how the large sites are handled on this page.

[mgifford]

So @harlan-zw  this should work to scan 500 URLs vs the default 200? It would also take 2 samples rather than just one.

Assuming this is in the directory where you execute the script:  unlighthouse.config.ts

export default {
scanner: {
// run lighthouse for each URL 2 times
samples: 2,
// increase the maximum number of routes - https://unlighthouse.dev/api/config#scannermaxroutes
maxRoutes: 500,
},
debug: true,
}