clients: use minimal 'url' polyfil instead of url-shim

[brendankenny]

Long ago (#5293) we substituted url-shim for the Node url module because robots-parser needed require('url').URL to give the WHATWG URL but the browserify polyfill for 'url' didn't have URL on it. In the meantime, the pubads plugin also started relying on that behavior, importing URL the same way. For some reason, the most popular rollup 'url' polyfills still don't have URL on them, so we've still been shoving url-shim in there.

However, there are other things on url-shim, and this can cause issues :)

We really only need the slimmest of polyfills for 'url', since only the URL property is needed, so this PR replaces it with one tiny version.

I also opened PRs in googleads/publisher-ads-lighthouse-plugin#325 and samclarke/robots-parser#23 to switch them both to the global URL, so we can remove this shim altogether when those land and are published.

[brendankenny]

this is only semi-related, it gets rid of the Circular dependency: polyfill-node.global.js -> polyfill-node.global.js warning when running yarn build-devtools (FredKSchott/rollup-plugin-polyfill-node#17). We still have the circular stream polyfill warnings though.

[brendankenny]

exporting globals is annoying. Open to nicer ways of doing this

[adamraine]

Hmm I was expecting this to cause issues in CI because we use fileURLToPath in root.js:

lighthouse/root.js

Line 24 in da73d4a

const dir = importMeta ? path.dirname(url.fileURLToPath(importMeta.url)) : __dirname;

[brendankenny]

That's what I was referring to in #13519 (comment), it doesn't work in the current version either. Either nothing is calling it or at least not calling it with an importMeta. If/when we do need that (if we don't transform it out of the bundle in a different way) we'll need to find a different polyfill because rollup-plugin-polyfill-node doesn't have fileURLToPath either :/

[paulirish]

robots parser already shipped a new version (3.0.0) with that PR. pubads didnt yet