Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document nfs home directories restore process #5286

Merged
merged 4 commits into from
Dec 16, 2024
Merged

Document nfs home directories restore process #5286

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
efeae0b
Document how to restore home directories from snapshot
sunu Dec 16, 2024
a5d60fa
We only have AWS specific documentation for now
sunu Dec 16, 2024
53f2131
Move the restore documentation to the right section
sunu Dec 16, 2024
987c6c2
Add an extra sub-heading
sgibson91 Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/howto/filesystem-management/filesystem-backups/restore-filesystem.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,42 @@ covers those steps for the cloud providers.
(howto:filesystem-backup:restore:aws)=
## AWS

### Restoring home directories from an EFS recovery point

```{note}
We follow AWS's guidance for [restoring EFS from a recovery point](https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-efs.html#efs-restore-console)
```

Due to the length of the steps listed in that document, we will not repeat them here.

### Restoring home directories from EBS snapshots

On AWS, the EBS volumes used for home directories are backed up using [Data Lifecycle Manager (DLM)](https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-lifecycle.html). This means that we can restore home directories from a snapshot if they are deleted or corrupted.

To restore a home directory from a snapshot, we need to create a new EBS volume from the snapshot, mount it to the EC2 instance attached to the existing EBS volume containing the NFS home directories, and then copy the contents from the restored EBS volume to the NFS home directories volume.

```{note}
Please follow AWS's guidance for [restoring EBS volumes from a snapshot](https://docs.aws.amazon.com/prescriptive-guidance/latest/backup-recovery/restore.html#restore-files) to create a new EBS volume from the snapshot.
```

Once we have created a new EBS volume from the snapshot, we need to mount it to the EC2 instance attached to the existing EBS volume containing the NFS home directories. To do this, we need to find the instance ID of the EC2 instance attached to the existing EBS volume. This involves the following steps:

1. Go to the EBS volumes page in the AWS console
2. Find the volume ID of the existing EBS volume containing the NFS home directories
3. Click on the volume ID and find the instance ID in the "Attached resources" section
4. Once we have the instance ID, we can mount the new EBS volume to the EC2 instance by following the steps outlined in the [Attaching an Amazon EBS volume to an instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html) guide.

Once we have mounted the new EBS volume to the EC2 instance, we can copy the contents from the restored EBS volume to the NFS home directories volume. This can be done by running the following commands on the EC2 instance:

```bash
# Copy the contents from the restored EBS volume to the NFS home directories volume
rsync -av --info=progress2 </restored-volume/path-to-home-directories/> </existing-volume/path-to-home-directories/>
Copy link
Member

@yuvipanda yuvipanda Dec 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs additional work, since we exclusively use EKS. It does create EC2 instances, but that's an implementation detail and we don't necessarily have packages in the base instance. I think we have to either:

  1. Create a new pod, new PVC and PV that refers the recovered volumeId, and mount the existing volume via NFS into that pod.
  2. Create a separate new EC2 instance (temporarily), and mount the recovered and existing home directory there to do a copy. However, we'll have to mount the existing home directory via NFS, which may be difficult to do outside the k8s cluster.

I think (1) is easier.

In addition, I think we shouldn't copy directly into the existing home directories, as that may delete new files that were created since the snapshot (default rsync behavior). So either:

  1. We are recovering everything (disaster recovery scenario), and we don't have access to the existing volume at all. So here we have to do a full copy
  2. We are recovering something partial, and we should copy into a subfolder and copy that into the users' home directory. We should do this sparingly.

```

Once we have copied the contents from the restored EBS volume to the NFS home directories volume, we can delete the EBS volume that was created from the snapshot.

After the restoration is complete, we should run `terraform plan` to double check that no unintended changes were made to the Terraform state.

(howto:filesystem-backup:restore:gcp)=
## GCP

Expand Down