Page is blocked from indexing (libUrl.parse is not a function)

[magtesyan]

Hi,

Get this fail on every website.
By the way clicking "Learn more" becomes 404.

[CymruUSA]

Same thing and yes, "Learn More" link returns a 404.

[patrickhulce]

looks like Audit Error: libUrl.parse is not a function is coming from robots-parser. robots-parser needs the url module and our gulpfile ignores it since we ship url-shim

https://github.com/samclarke/robots-parser/blob/322e6e3ff9868f921e8397a2f4026a047af0a02b/Robots.js#L179

@kdzwinel looks like we'll need to provide a shim for url that references our url-shim file instead, would you have bandwidth to take that on soon?

[kdzwinel]

@patrickhulce yep, it's definitely the parser I included. It doesn't have any external deps, but it uses 'url' and 'punycode' nodejs modules. It's my bad, sorry for that. I'll look at it ASAP. Should I pull that library into LH? Configure webpack to replace native modules with polyfills? Fork the library and make it work in both node and web environment? 🤔

[patrickhulce]

Should I pull that library into LH? Configure webpack to replace native modules with polyfills? Fork the library and make it work in both node and web environment? 🤔

ideal fix from my perspective would be adding a .parse method to our url-shim and seeing if there's a browserify way to have require('url') calls just use our shim instead rather than ignore. close 2nd would be using https://github.com/ds300/patch-package that @wardpeet found to patch it in LH repo to support node and web :)