FR: Allow external API calls

[rviscomi]

Feature request

I hear there's consideration to allow external API calls in audits. I wanted to create this FR to open up the discussion for anyone else who wants to follow along.

Summary

Rearchitect Lighthouse audits to be able to support async network calls like external APIs.

For example, in the JS vulnerability audit we're maintaining a version of the Snyk vulnerability DB. This was an intentional decision to keep the LH code self-contained and not rely on async API calls over the network. I'm curious if this guideline might be relaxed.

What is the motivation or use case for changing this?

There is no single open source project that validates structured data, similar to how aXe validates a11y. It may be more feasible to hook into a structured data validation API than to create a new open source project. This would allow the audit to provide rich suggestions and also not have to include any additional code dependencies.

How is this beneficial to Lighthouse?

This enables LH audits to make use of external projects like structured data, accessibility, or vulnerability data without requiring the projects to be open source, compatible, and bundled with LH. This could give us access to more high quality audits.

[patrickhulce]

Thanks @rviscomi! Just curious what prompted the "there's consideration to allow external API calls in audits" :)

Some of the counter-arguments for why we should require all functionality to be part of LH in no particular order:

• Having a copy of LH code and a fixed version of Chrome guarantees that you can reproduce results and run audits indefinitely. Adding a service into the mix is another point of failure and maintenance.
• Additional incentive to open source (or build an equivalent in the open) otherwise proprietary/closed systems thereby adding value to the entire ecosystem.
• Users can be confident their private data is not being beaconed somewhere without their knowledge/consent. (IMO any potential audits that do this would need to have some sort of consent mechanism and auto-opt-out like error reporting)
• Contacting an outside service is not an option in some environments (i.e. LR, offline, etc) and a parity gap in the default set of audits is undesirable.

[paulirish]

There is no single open source project that validates structured data, similar to how aXe validates a11y. It may be more feasible to hook into a structured data validation API than to create a new open source project

What's the gap between #4359 (or actually #5446 more specifically) and what you're describing?

[rviscomi]

Just curious what prompted the "there's consideration to allow external API calls in audits" :)

@kdzwinel mentioned that it came up in a recent eng meeting. I don't have much context.

Some of the counter-arguments for why we should require all functionality to be part of LH

No strong opposition to any of these points. All good things to keep in mind but I don't think any of them are necessarily prohibitive to the FR.

What's the gap between #4359 (or actually #5446 more specifically) and what you're describing?

Quite a large gap actually. Our initial SD audit focuses only on very basic validation of JSON-LD. The Structured Data Testing Tool for example handles more advanced validation of JSON-LD as well as other SD formats like Microdata and RDFa.

[paulirish]

Currently we dont see a great way to do this and support it across DevTools/CLI/ext/Lightrider/etc.

Complications

• Can they support the volume that we do?
• What is their caching? Can it handle very iterative dev
• User consent
• What about localhost auditing?
• long-term support and stability (how do they update the service/handle versioning)

Other options

1. link to the other tool with the URL prefilled
2. iframe the other tool's results page
3. open source the other tool so we can WASM it and run it.

[brendankenny]

In conclusion, we want to remain local only on the Lighthouse side, though a specific compelling case may someday change that.

However, plugins open an opportunity for non-local services to be used. Plugins are just node modules and so can do what they wish, including connecting to external APIs, and so could provide extra processing or feedback (e.g. calling out to the Structured Data Testing Tool).