Creating and Restoring from Backup

Overview

Codalab has a custom command written for it that creates a database backup, and copies it to the storage you are using under /backups. Here's we'll see how to execute and automate that command, and how to restore from one of these backups in the event of a failure.

Creating Backups

There's a custom command on v2 that we use to create, and upload database backups to storage. It should be accessible from inside the Django container (docker-compose exec django bash) with python manage.py upload_backup. It optionally takes an argument backup_path which is the path relative to your storage bucket, /backups. For instance if I pass it as 2019, the backup should goto the folder /backups/2019 in your storage bucket.

Scheduling Automatic Backups

To schedule automatic backups, we're going to schedule a daily cronjob. To start, open the cron editor in a shell with crontab -e.

Add a new entry with our command being docker exec django python manage.py upload_backup <Optional upload_backup argument>. Since we want this to run daily it may look like so:

0 0 * * * docker exec django python manage.py upload_backup

This specifies that at midnight (0m, 0h) for any day, for any month, for any day of the week we run this command.

You may notice we're using plain docker here and not docker-compose and that's because for docker-compose to work, we have to run the command from the same directory as docker-compose.yml. Instead we use docker and reference the full container name.

You should also en-sure the command you're using for your cronjob works before adding it in the cron editor.

Once done, save and quit the crontab editor, and verify your changes held by listing out cronjobs with crontab -l. You should see your new crontab entry.

Restoring From Backup

Re-install Codalab according to the documentation here:

Once Codalab is re-installed and working, we're ready to restore our database backup. Upload the database backup to the webserver. It should go under the competitions-v2 install folder in the /backups directory. For example your path might look like: /Users/ubuntu/competitions-v2/backups

Once the backup is located in the /backups folder, you'll want to get into the postgres container (docker-compose exec postgres bash). Make sure you know your DB_NAME, DB_USER, and DB_PASSWORD variables from your .env.

You can restore two ways. The first would be manually dropping the db, re-creating it, then using pg_restore to restore the data:

container$ dropdb $DB_NAME -U $DB_USER
container$ createdb $DB_NAME
container$ pg_restore -U $DB_USER -d $DB_NAME -1 /app/backups/<filename>.dump

Or, you can let pg_restore know to do that for you with a couple of flags/arguments:

pg_restore --verbose --clean --no-acl --no-owner -h $DB_HOST -U $DB_USER -d $DB_NAME /app/backups/<filename>.dump

The arguments --no-acl & --no-owner may be useful if you're restoring as a non-root user. The owner argument is used for: Do not output commands to set ownership of objects to match the original database.

The ACL argument is for: Prevent dumping of access privileges (grant/revoke commands).

After running pg_restore successfully without errors, you should find everything has been restored to before the data loss.