[EPIC] Setup a system for automated GCP filestore backups

[yuvipanda]

Part of #4390

The Problem

Unlike AWS and Azure, GCP doesn't have a way to automatically schedule backups of Filestore. Google's own suggested solution is pretty halfbaked, and seems to be focused on 'how many GCP specific features can we throw together?'. The python code they provide to do backups is particularly bad, with no error handling nor retention policies.

Scope

Filestore has been the least nice of all managed home directory systems. My suspicion is that GCP is aware of this massive missing piece, and hopefully within a few years we will have an automated solution. So the goal here is to focus on doing the least possible we can to make sure we have some backups we can trust.

Tasks

Beta Give feedback

1. Create a python script (gcp-filestore-backups) to perform backups and retention of filestore instances #4393
   +0
   
2. Create a docker image for the GCP filestore script to run in #4394
   +0
   
3. Create a deployment and service accounts in the support/ chart for gcp-filestore-backup to run #4395
   +0
   
4. Create cloud sevice accounts with permissions needed for gcp-filestore-backup to run #4396
   +0
   
5. Deploy gcp-filestore-backup to the 2i2c shared cluster on GCP #4397
   +0
   
6. Allow gcp-filestore-backups to support multiple filestores #4449
   +0
   
7. [Spike] [2h] Investigate how we can alert if a backup has not been performed #4536
   +0
   
8. Design a manual process for verifying GCP backups #4611
   +0
   
9. Deploy filestore backups to all GCP clusters #4683
   +0
10. [EPIC] Separate out gcpFilestoreBackups from 2i2c support chart into it's own helm chart #4664
    5 of 5
    +0

[yuvipanda]

@sgibson91 what's left to get this marked as completed? Deployment to our various GCP clusters?

[sgibson91]

Yes. I feel like there was an extra "communicate to communities" step at one point that seems to have disappeared from the task list?