Merge pull request #4459 from sgibson91/howto-recover-filesystems

Document enabling backups of and recovering filestores on GCP

docs/howto/filesystem-backups/enable-backups.md

@@ -0,0 +1,63 @@
+(howto:filesystem-backups:enable)=
+# Enable Automatic Filesystem Backups
+
+This document covers how to enable automatic filesystem backups across the cloud
+providers we use.
+
+(howto:filesystem-backups:enable:gcp)=
+## GCP
+
+```bash
+export CLUSTER_NAME=<cluster-name>
+```
+
+1. **Create relevant resources via terraform.**
+
+   Our terraform configuration supports creating the relevant resources to support
+   automatic filesystem backups, including: creating a GCP IAM Service Account
+   with enough permissions to manage backups, binding that Service Account to
+   a Kubernetes Service Account, and outputting the relevant annotation to use
+   in helm chart config to make the relevant connections.
+
+   1. In `terraform/gcp/projects/<cluster-name>.tfvars`, add the following variable:
+      ```
+      enable_filestore_backups = true
+      ```
+   1. Ensure you are in the correct terraform workspace to apply this change:
+      ```
+      terraform workspace select $CLUSTER_NAME
+      ```
+   1. Plan and apply the changes
+      ```bash
+      terraform plan -var-file=projects/$CLUSTER_NAME.tfvars
+      terraform apply -var-file=projects/$CLUSTER_NAME.tfvars
+      ```
+
+1. **Enable the `gcpFilestoreBackups` deployment in the cluster's support values.**
+
+   1. In `config/clusters/<cluster-name>/support.values.yaml`, add the following config:
+      ```yaml
+      gcpFilestoreBackups:
+        enabled: true
+        filestoreName: <filestore-name>
+        project: <gcp-project>
+        zone: <gcp-zone>
+        annotations:
+          iam.gke.io/gcp-service-account: <gcp-service-account-email>
+      ```
+      where:
+      - `filestoreName` is the name of the filestore to be backed up (can be found
+        from the Filestore Instances page in the GCP console)
+      - `project` is the name of the GCP project in which the filestore exists
+      - `zone` is the GCP zone the filestore is deployed to and where the backups
+        will be stored (e.g. `us-central-b`)
+      - `annotations` is the output from the `terraform apply` command in the
+        previous step. You can run `terraform output gcp_filestore_backups_k8s_sa_annotations`
+        to retrieve this.
+1. **Upgrade the support chart.**
+   ```bash
+   deployer deploy-support $CLUSTER_NAME
+   ```
+
+This will have successfully enabled automatic backups of GCP Filestores for this
+cluster.

docs/howto/filesystem-backups/index.md

@@ -0,0 +1,13 @@
+(howto:filesystem-backups)=
+# Filesystem Backups
+
+The filesystem is where our users keep a lot of critical files, such as code.
+Hence if the filesystem is lost, for some reason, this is a disastrous event.
+These documents describe how to enable automated backups of the filesystems for
+various cloud providers, and recover filesystems from those backups.
+
+```{toctree}
+:maxdepth: 2
+enable-backups
+restore-filesystem
+```

docs/howto/filesystem-backups/restore-filesystem.md

@@ -0,0 +1,23 @@
+(howto:filesystem-backup:restore)=
+# Restore a Filesystem from a Backup
+
+In the event of a disaster and the filesystem needs recovering, this document
+covers those steps for the cloud providers.
+
+(howto:filesystem-backup:restore:gcp)=
+## GCP
+
+```{note}
+We follow GCP's guidance for [restoring fileshares from a backup](https://cloud.google.com/filestore/docs/backup-restore#restore)
+```
+
+To restore a share on a Filestore instance on GCP, we follow the documentation
+linked above. In short, this involves:
+
+1. [Go to the Filestore instances page](https://console.cloud.google.com/filestore/instances) in the GCP console
+1. Click the instance ID of the Filestore you want to restore and click the "Backups" tab
+1. Locate the backup you want to restore from (most likely the most recently created), and click (...) "More actions"
+1. Click "Restore backup" and then select "Source instance"
+1. Click "Restore" and complete the dialog box that appears
+
+This should successfully restore the Filestore instance to its last backed-up state

docs/index.md

@@ -74,6 +74,7 @@ howto/troubleshoot/index.md
 howto/regenerate-smce-creds.md
 howto/budget-alerts
 howto/decrease-size-gcp-filestore
+howto/filesystem-backups/index
 ```
 
 ## Topic guides