VOTE-972: Add robotstxt module. Delete default robots.txt file and pr…

[daniel-crowder]

…event if from being committed to repo.

Jira ticket

VOTE-972

Description

Add contrib module, RobotsTxt, and configure the robots.txt out put for production's static site.

Deployment and testing

Post-deploy steps

1. Update your setting.local.php file to set config split environment to production
   $config['config_split.config_split.production']['status'] = TRUE;
2. Locally run lando retune

QA/Testing instructions

1. Goto /robots.txt and confirm the output contains a list of Allows and Disallows for the environment(enabled config split setting).
2. Run lando static to generate static site locally.
3. Confirm that the robots.txt and the correct entries are found in the static directory.
4. Update your setting.local.php file to set config split environment to non_production
   $config['config_split.config_split.non_production']['status'] = TRUE;
5. Run lando drush cim to import non-production configs.
6. Goto /robots.txt and confirm the output is set to block everything from being crawled.
7. Repeat Steps 2-3
8. Repeat Steps 4-7 after updating config split setting to
   $config['config_split.config_split.local']['status'] = TRUE;

Checklist for the Developer

• A link to the JIRA ticket has been included above.
• No merge conflicts exist with the target branch.
• Automated tests have passed on this PR.
• A reviewer has been designated.
• Deployment and testing steps have been documented above, if applicable.

Checklist for the Peer Reviewers

• The file changes are relevant to the task objective.
• Code is readable and includes appropriate commenting.
• Code standards and best practices are followed.
• QA/Test steps were successfully completed, if applicable.
• Applicable logs are free of errors.

[rayestrada]

Will need to wait until launch week to deploy this one.