Consider adding a nightly database dump

[nonprofittechy]

Although Postgres is transaction safe, this would still improve recoverability. We may want to keep at least a few days of dumps as well.

[BryceStevenWilley]

Some additional thoughts:

• This could be as simple as a cron job running on the AWS server or in the postgres container that runs pg_dump ¹, and sends the gzipped dump to S3.
• Alternatively, we might be able to run pg_dump as a scheduled job from the java size? idk.
• One more alternative is to try to directly backup the docker volume where the postgres data is stored. This might not be optimal, as it wouldn't work across postgres versions I think. https://www.howtogeek.com/devops/how-to-back-up-your-docker-volumes/

Footnotes

1. Of the options mentioned in the postgres docs, the Write ahead log (WAL) is mostly for crash protection, and taking a file level archive says it should only be done when postgres is not running. ↩