Robots.txt deploy for domserver on online instance

[vmcj]

As suggested by @meisterT, we should deploy a robots.txt to prevent spiders from triggering Symfony errors when they visit our POST only routes.

[eldering]

Could we include that by default in the domjudge repo itself? I guess you'd never want this behaviour.

[vmcj]

The traditional robots.txt will not work when you deploy DOMjudge in a subdirectory (https://stackoverflow.com/q/4837334)
So we can either set a <meta> or a header, see: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Because I prefer to not fill the DOM with extra tags if not needed I think we should do this with another header (and copy this manually to the domjudge-scripts repo).