The main things we want indexed are record pages themselves (info request pages, user pages, authority pages, etc).

Snippets of request content often appear on list pages, and create a whack-a-mole situation when unhappy users find that external search engines have indexed a list page (e.g. /body/foo?page=12) that contains a cached snippet of PII that we've removed from the request page itself.

We should stop indexing of:

• Request list pages (/list/all, /list/successful, etc) with a page= query param
• Similar requests page (/request/:url_title/similar)
• Body pages with a page= query param (/body/:url_name?page=N)
• User pages with a page= query param (/user/:url_name?page=N)
• User "wall" page (/user/:url_name/wall)

We might be able to do this via robots.txt, or could set via the X-Robots-Tag header depending on the page number:

before_action :set_no_crawl_headers, if: -> { params[:page].to_i > 1 }