core(is-crawlable): make sure that page is not blocked by robots.txt file #4548
Conversation
| return response.text() | ||
| .then(content => ({status: response.status, content})); | ||
| }) | ||
| .catch(_ => ({status: null, content: null})); |
There was a problem hiding this comment.
Second part of #4356 is a new audit that checks if robots.txt file is valid. That audit should fail if fetch returns HTTP 500+ and that's why we collect it.
|
Also... looking at the robots-parser test suite, do you think there are any additional cases they should add? i'm looking at https://github.com/ChristopherAkroyd/robots-txt-parser/tree/master/test/test-data https://github.com/python/cpython/blob/master/Lib/test/test_robotparser.py https://github.com/meanpath/robots/tree/master/test |
|
@paulirish we do show where the blocking directive comes from (header, meta, robots.txt) - check out the screenshot at the top. I'd be great to show which robots.txt directive is blocking, but Regarding tests - I quickly scanned test suites that you linked and |
First part of #4356
Extends is-crawlable audit, so that it also verifies robots.txt file (besides meta tag and header).
This change adds a new dependency - robots-parser. It's a lightweight, well tested robots.txt parser with no dependencies.