Search engine crawlers see scripts at the extensionless endpoints

[stuartpb]

We need a "user-agent-is-robot" or similar to distinguish requests from search engines so they seem a page similar to what browsers see (without the uasniffed banner, though, since we don't want that messing with search results).