[SEO Audits] Page is not blocked from indexing

[rviscomi]

Audit group: Crawling and indexing
Description: Page is not blocked from indexing
Failure description: Page is blocked from indexing
Help text: The “Robots” directives tell crawlers how your content should be indexed. Directives like noindex and none prevent indexing. Learn more about the Robots meta tag and header.

Success conditions:

• No tag matching the query selector meta[name=robots] has a content attribute with a comma-separated value in the blocklist below
• X-Robots-Tag response header does not have a value in the blocklist below

Blocklist:

• noindex
• none

Notes:

• Site owners can block crawlers via: meta tags, robots.txt rules, or X-Robots-Tag response header.
  • In next version, check that location.href is not disallowed by robots.txt
• The robots meta name attribute is interchangeable with other bot names including:
  • msnbot
  • bingbot
  • google
  • yandex

[kdzwinel]

Edge cases and my current strategy to deal with them:

1. NOINDEX - not sure if uppercase/mixedcase is supported - assuming it's valid
2. noindex nofollow - not sure if space (instead of coma) is supported - assuming it's not valid
3. noindex, nofollow, all - not sure if all overrides previous restrictions - assuming it's not valid
4. unavailable_after - not supporting
5. <meta name="googlebot" - not supporting

@rviscomi let me know if'd like to make any changes.

[rviscomi]

1. NOINDEX - not sure if uppercase/mixedcase is supported - assuming it's valid

👍

2. noindex nofollow - not sure if space (instead of coma) is supported - assuming it's not valid

👍

3. noindex, nofollow, all - not sure if all overrides previous restrictions - assuming it's not valid

Let's fail as long as we see anything in the blocklist.

4. unavailable_after - not supporting

Let's try to support this. Fail the audit if the date is in the past, otherwise provide some kind of warning that the page will not be indexable soon.

5. meta name="googlebot" - not supporting

👍