doc.rust-lang.org/robots.txt doesn't match src/doc/robots.txt

[carols10cents]

The intention in #59213 was to add robots.txt to what gets uploaded to doc.rust-lang.org, but it appears that either the manual step of copying this over for a release hasn't been happening, or that the automation was never done, or the automation isn't working.

The contents of src/doc/robots.txt doesn't match what I'm currently seeing on https://doc.rust-lang.org/robots.txt:

User-agent: *
Disallow: /0.3/
Disallow: /0.4/
Disallow: /0.5/
Disallow: /0.6/
Disallow: /0.7/
Disallow: /0.8/
Disallow: /0.9/
Disallow: /0.10/
Disallow: /0.11.0/
Disallow: /0.12.0/
Disallow: /1.0.0-alpha/
Disallow: /1.0.0-alpha.2/
Disallow: /1.0.0-beta/
Disallow: /1.0.0-beta.2/
Disallow: /1.0.0-beta.3/
Disallow: /1.0.0-beta.4/
Disallow: /1.0.0-beta.5/

(lines 20-27 in the repo are not present on the live site)

Thanks!

[kornelski]

Still not up to date after 1.42 release: https://doc.rust-lang.org/robots.txt

[Mark-Simulacrum]

AFAICT, the relevant change has not yet made it into stable, but is in beta and nightly. I expect that means that we should get the robots.txt file populated appropriately in roughly 6 weeks. Closing this as fixed for now, if it's still wrong after 1.43 is released we can reopen (though ideally on rust-central-station, and not here, as remaining work should happen there).

[steveklabnik]

Closing this as fixed for now, if it's still wrong after 1.43 is released we can reopen (though ideally on rust-central-station, and not here, as remaining work should happen there).

It seems like this fix didn't work after 1.43; I'm not as familiar with rust-central-station. What's the procedure for moving forward here?

[Mark-Simulacrum]

I've filed an issue over there (rust-lang/rust-central-station#719) and will try to investigate a bit right now, it looks like it's hopefully a fairly easy fix.