Enable automated backups of EBS volumes on VEDA staging hub

[sgibson91]

Context

We will be moving forward on automated backups for AWS EBS volumes using the Data Lifecycle Manager option.

• Automatically create daily backups
• Retention period of 5 days
• Deploy and test on VEDA staging hub first

Task list

Tasks

Preview Give feedback

1. Create an IAM role for DLM service in terraform
2. Apply the target tag to existing EBS volumes for which automated backup has to be enabled.
3. Set up a DLM lifecycle policy, that defines the time and interval of backup creation, retention period and a target tag for selection of EBS volumes for backing up, in terraform
4. Set Backup = true on the existing EBS volume for staging
5. Wait a day and see if a backup has been created
6. Investigate how to restore from backups
7. Document feature and how to restore from backup

Definition of Done

• The feature/service is technically complete
• Deployed to staging hub
• The feature/service been tested with one or more users (if applicable)
• The feature/service is well documented, and the documentation is accessible for the target user base

[sgibson91]

#5189 applied successfully!

The policy will execute a backup snapshot at 23:45 UTC every day, so next items are:

• Check back tomorrow to see if a backup snapshot of the staging EBS volume has been created
• Check back in >5 days time to ensure we are only retaining 5 backup snapshots for the EBS volume

[sgibson91]

AWS's docs on restoring EBS volume from a snapshot https://docs.aws.amazon.com/prescriptive-guidance/latest/backup-recovery/restore.html#restore-snapshot

[sgibson91]

Gotcha: double check terraform state hasn't changed state due to the backup, see for example the GCP backup docs https://infrastructure.2i2c.org/howto/filesystem-backups/restore-filesystem/#gcp

[sgibson91]

Woohoo! Another piece completed!

[yuvipanda]

Great job, @sgibson91 and @sunu!

Unfortunately I don't think this is quite complete yet. I've left a comment in https://github.com/2i2c-org/infrastructure/pull/5286/files#r1887447565.

[sgibson91]

Did we close this in the end? @yuvipanda @sunu

We definitely updated the deployer to create a PV and PVC with the volume hosting the snapshot attached to do a copy

[yuvipanda]

@sgibson91 i am not sure! once either of you verify (via the aws console maybe?) feel free to close this if appropriate.

[sgibson91]

I think the comment you left on the PR was more about the docs to do recovery. But I will certainly click around in the console to ensure back-ups are enabled next week.

[sunu]

@yuvipanda @sgibson91 We have automated snapshots for both VEDA staging and prod hubs:

And as Sarah mentioned, we have updated the deployer CLI to create a PV and PVC with the volume hosting the snapshot attached to do a copy and updated the documentation accordingly in #5291

I think we can close this issue as completed.