Update robots.txt

[cqueern]

Updated robots.txt so well-behaved search engine crawlers will not index Crossfeed instances, as I assume they're not meant to appear in search engine results due to the potentially sensitive information they collect.

🗣 Description

💭 Motivation and context

🧪 Testing

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• Changes are limited to a single goal - eschew scope creep!
• All future TODOs are captured in issues, which are referenced
  in code comments.
• All relevant type-of-change labels have been added.
• I have read the CONTRIBUTING document.
• These code changes follow cisagov code standards.
• All relevant repo and/or project documentation has been updated
  to reflect the changes in this PR.
• Tests have been added and/or modified to cover the changes in this PR.
• All new and existing tests pass.

✅ Pre-merge checklist

• Revert dependencies to default branches.
• Finalize version.

✅ Post-merge checklist

• Create a release.

[Matthew-Grayson]

@cqueern Thanks for taking an interest in Crossfeed! You bring up an interesting point about whether or not we want to be indexed. I believe that allowing indexing was a deliberate decision, but I will look into it further.

[cqueern]

My pleasure! It's an important project.

Sounds good. If the decision is to exclude all robots from the entire server, suggest that line 3 in the robots.txt file remain

Disallow: /

If the decision is to allow all robots complete access, suggest that line 3 in the robots.txt file be updated to

Disallow: