Rewrite backup routine

[warrensearle]

We need to rewrite our backup routine code. See screen grab below:

Task list

• Update googleapis SDK to the latest version. (from 49 to 67)
• Check backups are working after changes

Research

The current googleapis SDK is 18 versions behind the latest stable one. Google has deprecated v4 and v5. As seen in the screenshot above, we're currently on v4 of the SDK (49.0.0). Having that in mind we need to update our dependency to at least 67.0.0 to no longer face deprecation issues / warnings.

• Identify any third-party plugins that may use old versions (that call outdated API endpoints - Google).

• Check the latests best practices and any improvements we can add to our backup routine. (The backup routine currently in place is a great solution. It's also recommended by Firebase).

[KateMJAC]

Assigned to Halcyon in the March WO.

[HalcyonJAC]

I checked the Cloud Storage being used on the Google account and backups appear to be taken at 23:00, dispite the error that appears in the logs.

Tested existing backupFirestone() function locally (on the develop database) which appears to work fine.

Upgraded googleapis module to latest version and retested backup function - still works fine.

[HalcyonJAC]

To elaborate on the above, the tests done locally aren't for the scheduled backup job itself as that runs on Google cloud; we won't know if upgrading the googleapis module fixes the overnight backup problem until this change is merged into develop (then wait 24 hours). If it does not fix the problem though, the next step I think would be to upgrade the firebase/firestone modules, which would mean a lot of back-testing!

Note: From what I could see in the Firebase account the overnight backups (i.e. the for production database) appear to be taking place dispite the error message that appears in the firebase logs. Perhaps the error is a false negative coming back from an old API.

Note that Firebase also has an automated backup option that can be set via Firebase control panel itself, compressed using gzip and (optionally) deleted after 30 days, rather than the backups being done via code. I don't know if that option has been considered...?

[warrensearle]

Hey @HalcyonJAC if there's now an automated backup option via the control panel then that would be awesome and the preferred solution!! I know there're backups for Realtime Database but haven't spotted the equivalent for Firestore or Authentication.

If we need to continue with the current approach please let me know and I'll continue review/testing this PR.

[HalcyonJAC]

No, your're right there isn't an automated backup function available via the Firebase Console for the Authentication database.

So, assuming you want to keep that going then, I guess we better proceed with the PR.

Thanks!

[HalcyonJAC]

Blocked by #295 and #473

Note: A few weeks ago this ticket was put on hold as @warrensearle has/had another ticket open where he was altering the code in the backup routines for some reason.

Once @warrensearle has finished updating the backup routine, this ticket can be resumed

[KateMJAC]

@HalcyonJAC can this ticketd be progressed yet?

[warrensearle]

Hey @JeremykJAC and @HalcyonJAC sorry for the delay getting back to you about this ticket.

I’ve tidied up the firestore backup routine in the following PR:
#578

Plus have configured permissions so backups now work on develop environment and also can be run from local node environment (using nodeScript).

A similar code tidy is now needed for the authentication backup routine so I’m hoping you can sort that as part of finishing off ticket 403 - you can add to the above PR.

Good luck 😄