[feature] Beef up our AI opt-outs

[daenney]

Description

• Updates our robots.txt with upstream
• Add two headers that a number of AI crawlers respect as an opt-out of their crap

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• I/we have read the GoToSocial contribution guidelines.
• I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
• I/we have not leveraged AI to create the proposed changes.
• I/we have performed a self-review of added code.
• I/we have written code that is legible and maintainable by others.
• I/we have commented the added code, particularly in hard-to-understand areas.
• I/we have made any necessary changes to documentation.
• I/we have added tests that cover new code.
• I/we have run tests and they pass locally with the changes.
• I/we have run go fmt ./... and golangci-lint run.