Merge pull request #3997 from rust-lang/robots-infra

infra: add robots.txt for GH Pages previews

Author: chriskrycho

ADMIN_TASKS.md

@@ -133,3 +133,13 @@ $ dot dot/trpl04-01.dot -Tsvg > src/img/trpl04-01.svg
 In the generated SVG, remove the width and the height attributes from the `svg`
 element and set the `viewBox` attribute to `0.00 0.00 1000.00 1000.00` or other
 values that don't cut off the image.
+
+## Publish a preview to GitHub Pages
+
+We sometimes publish to GitHub Pages for in-progress previews. The recommended
+flow for publishing is:
+
+- Install the `ghp-import` tool by running `pip install ghp-import` (or `pipx install ghp-import`, using [pipx][pipx]).
+- In the root, run `tools/generate-preview.sh`
+
+[pipx]: https://pipx.pypa.io/stable/#install-pipx

tools/generate-preview.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+mdbook build
+cp ./tools/preview-robots.txt ./book/robots.txt

tools/preview-robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /