Provision new machine

Note: Guidelines in here

Terraform

• Use terraform to provision new machine.
  • ./build.rb create <machine> to start new terraform stack.
  • Review variables.tf file.
    • Make sure use_ansible is false to start with.
    • Change variable has_backup to true if the machine should have backups, and uncomment outputs.tf. Check Backups Strategy for details.
    • Check the DNS aliases. Each application/port publicly accessible should be using a different DNS record, as we aim to only use port 443 for all our web applications.
    • Run terraform using ./build.rb plan <machine> and ./build.rb apply <machine>
    • Update documentation:
      • ./build.rb docs
      • ./build.rb plan docs && ./build.rb apply docs

At this point, machine will be only accessible using root/terraform private key

If the ./build.rb apply <machine> command produces timeout errors, first confirm if you're able to ssh to the machine by the running the next command. A similar timeout error might be caused by ssh port 22 being blocked by the ISP. Try connecting to a different network and run terraform again or connect manually.

ssh -i conf/provisioning/ssh/terraform-api.key ubuntu@<server>.openmrs.org

Ansible (first run)

• Add machine to relevant inventory file and create host var files in ansible.
• If the machine should have a docker apps, follow this guide to check all requirements
• Verify that datadog tags have the correct values.
• If the machine should have backups, add to backups group and add outputs from terraform as aws variables in host_var file.
• Secrets should go to vault file, and should be encrypted using ansible-vault
• Commit and push changes to github

To run ansible you can either:

a) Modify terraform variable use_ansible to true, and run ./build.rb plan <machine> and ./build.rb apply <machine>. For some reason, the first runs can fail on datadog

or

b) Run ansible locally with: $ ansible-playbook -vv -i inventories/prod-tier<> --limit <machine>.openmrs.org --user ubuntu --private-key ~/<path-to-terraform-repo>/conf/provisioning/ssh/terraform-api.key site.yml

Running terraform after running ansible

In certain cases, terraform will attempt to ssh (as root) into machines. When ansible runs, it disables root SSH access.

To run terrafomr, ssh into the machine and temporary enable Root SSH access:

$ sudo vi /etc/ssh/sshd_config

# edit line
PermitRootLogin No
# to 
PermitRootLogin Yes

$ service sshd restart

After running terraform, undo this manual change.

Monitoring

• Verify alerts in datadog
• Add monitoring in pingdom and status.io (only if production)
• Add machine to Server inventory and add its service to this wiki