feat(http): Add X-Robots-Tag header

[bsmth]

Description

This PR adds the X-Robots-Tag header page.

Omitting historical rules no longer used by Google, at least

• noarchive
• nocache
• nositelinkssearchbox

See https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#directives

Motivation

We're missing docs about it, and it's a widely used header for controlling indexing & search presentation.

Related issues and pull requests

• Missing HTTP header-specific pages (known) #1458

[github-actions]

Preview URLs

• /en-US/docs/Glossary/Robots.txt
• /en-US/docs/Web/HTML/Element/meta/name
• /en-US/docs/Web/HTTP/Headers/X-Robots-Tag

External URLs (3)

URL: /en-US/docs/Web/HTTP/Headers/X-Robots-Tag
Title: X-Robots-Tag

• https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag (2 times) (Note! This may be a new URL 👀)

---

URL: /en-US/docs/Glossary/Robots.txt
Title: Robots.txt

• https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt (1 time) (Note! This may be a new URL 👀)
• https://www.robotstxt.org (1 time) (Note! This may be a new URL 👀)

(comment last updated: 2024-12-11 12:58:12)

[estelle]

I know very little about HTTP headers and knew very little about robot directives, so I had many questions. I realize this is a reference and not a guide, so you may want to disregard some of my questions in favor of a guide.

it was really good, and I learned a lot. In addition to the above clarifications, just a few suggestions to make it less wordy for ease of reading.

[estelle]

do we have a guide on setting HTTP headers that we can link to?

[bsmth]

Not yet, but that's a good idea. We could have a how-to page for this in the HTTP section, it might be good to talk about cases where you have a website but can't control headers (GitHub pages?)

[estelle]

I got a notification so re-reviewed. I just realized you didn't ask for another review yet, but already reviewed. as some of it may be helpful, sharing.

[bsmth]

I got a notification so re-reviewed. I just realized you didn't ask for another review yet, but already reviewed. as some of it may be helpful, sharing.

Super, thank you. I haven't incorporated all of the suggestions yet, so I can re-request when I think everything's in a good state 👍🏻

[estelle]

Looking really, really good. Just think we may need a note on the all on one line example to indicate that the directives for all user agents goes first, and then exceptions for individual bots goes after if doing on one line (if that is accurate).

[pepelsbey]

Looks good! Thanks :)

Adding just one nitpick to the review. I’d suggest calling <meta> an element, not a tag. Unlike other sources do 🥲 It’s called an element in the spec, and we do it mostly consistently throughout the MDN.

Here’s how I usually explain the difference. The <p> element consists of opening <p> and closing </p> tags. Element is a language entity with its semantics, attributes, etc. Tag is a syntactic construction that makes all elements work.

[bsmth]

Thanks, both. I think this is ready to merge in case you want to have another quick look