docker-postgres-s3

A Docker-based solution for Postgres backups/restores on Amazon S3.

Goals

This project solves the following problems:

• I need a Postgres backup/restore solution which uses S3
• I want this solution to run in a container
• I want this container to be separate from the actual Postgres database
• I want to trigger the backups by HTTP, so I can use any scheduling mechanism I want
• I want to easily trigger restores as part of my development cycle

Usage

Instantiate the image with a target postgres database and an existing S3 bucket. The container will have a service running on port 8000, with the following endpoints:

• GET http://service:8000/status/health/ - a simple health check
• POST http://service:8000/backup/ - create a new backup and copy this to S3
• POST http://service:8000/restore/ - obtain the backup from S3 and restore it into the target database

Environment variables

Name	Required	Description
POSTGRES_DB	No, defaults to postgres	Name of the Postgres database to back-up from/restore into.
POSTGRES_USER	No, defaults to the value of POSTGRES_DB	Name of the Postgres user to connect as.
POSTGRES_PASSWORD	Yes	Password to use when connecting to the database.
S3_ENDPOINT	Yes	S3 endpoint to use when connecting. See http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for a list of valid endpoints.
S3_REGION	Yes	S3 region containing the bucket
S3_BUCKET	Yes	Name of the S3 bucket to store the database dump
S3_FILE	No	Name of the S3 file. Defaults to ${POSTGRES_DB}.pgdump
AWS_ACCESS_KEY_ID	Yes	AWS Access key. For uploads, requires write permissions on the bucket.
AWS_SECRET_ACCESS_KEY	Yes	AWS Secret key.
PG_RESTORE_ARGS	No	Extra arguments to pass to pg_restore.

Examples

The following examples are all shown as docker-compose files.

100% Docker, minimal example

Here the target database is also managed by docker.

version: "2"
services:
  db:
    image: postgres:9.5
    environment:
      POSTGRES_PASSWORD: password
      
  postgres-s3:
    image: publysher/postgres-s3
    links:
        - "db:database"     # target database should always be linked as `database`
    ports:
        - "8000:8000"       # you should not expose this to a public network
    environment:
      POSTGRES_PASSWORD: password
      S3_ENDPOINT: "https://s3.amazonaws.com"
      S3_REGION: "us-east-1"
      S3_BUCKET: "s3://my-bucket"
      AWS_ACCESS_KEY_ID: "access-key"
      AWS_SECRET_ACCESS_KEY: "secret"

Using an external database

In this example, the external database is not managed by docker. We use the extra_hosts option to connect to the database.

version: "2"
services:
  postgres-s3:
    image: publysher/postgres-s3
    environment:
      POSTGRES_DB: my_database
      POSTGRES_USER: my_user
      POSTGRES_PASSWORD: password
      S3_ENDPOINT: "https://s3.amazonaws.com"
      S3_REGION: "us-east-1"
      S3_BUCKET: "s3://my-bucket"
      AWS_ACCESS_KEY_ID: "access-key"
      AWS_SECRET_ACCESS_KEY: "secret"
    extra_hosts:
        - "database":192.168.99.101

Testing

Use docker-compose build && docker-compose run --rm tests to run the integration tests.