This will build a container for backing up multiple types of DB Servers
Currently backs up CouchDB, InfluxDB, MySQL, Microsoft SQL, MongoDB, Postgres, Redis servers.
- dump to local filesystem or backup to S3 Compatible services, and Azure.
- select database user and password
- backup all databases, single, or multiple databases
- backup all to separate files or one singular file
- choose to have an MD5 or SHA1 sum after backup for verification
- delete old backups after specific amount of time
- choose compression type (none, gz, bz, xz, zstd)
- connect to any container running on the same system
- Script to perform restores
- Zabbix Monitoring capabilities
- select how often to run a dump
- select when to start the first dump, whether time of day or relative to container start time
- Execute script after backup for monitoring/alerting purposes
- github.com/tiredofit/docker-db-backup
NOTE: If you are using this with a docker-compose file along with a seperate SQL container, take care not to set the variables to backup immediately, more so have it delay execution for a minute, otherwise you will get a failed first backup.
- You must have a working connection to one of the supported DB Servers and appropriate credentials
Clone this repository and build the image with docker build <arguments> (imagename) .
Builds of the image are available on Docker Hub
Builds of the image are also available on the Github Container Registry
docker pull ghcr.io/tiredofit/docker-db-backup:(imagetag)The following image tags are available along with their tagged release based on what's written in the Changelog:
| Alpine Base | Tag |
|---|---|
| latest | :latest |
docker pull docker.io/tiredofit/db-backup:(imagetag)Images are built primarily for amd64 architecture, and may also include builds for arm/v7, arm64 and others. These variants are all unsupported. Consider sponsoring my work so that I can work with various hardware. To see if this image supports multiple architectures, type docker manifest (image):(tag)
-
The quickest way to get started is using docker-compose. See the examples folder for a working docker-compose.yml that can be modified for development or production use.
-
Set various environment variables to understand the capabilities of this image.
-
Map persistent storage for access to configuration and data files for backup.
-
Make networking ports available for public access if necessary
The following directories are used for configuration and can be mapped for persistent storage.
| Directory | Description |
|---|---|
/backup |
Backups |
/assets/scripts/pre |
Optional Put custom scripts in this directory to execute before backup operations |
/assets/scripts/post |
Optional Put custom scripts in this directory to execute after backup operations |
This image relies on an Alpine Linux base image that relies on an init system for added capabilities. Outgoing SMTP capabilities are handled via msmtp. Individual container performance monitoring is performed by zabbix-agent. Additional tools include: bash,curl,less,logrotate, nano.
Be sure to view the following repositories to understand all the customizable options:
| Image | Description |
|---|---|
| OS Base | Customized Image based on Alpine Linux |
| Parameter | Description | Default |
|---|---|---|
BACKUP_LOCATION |
Backup to FILESYSTEM, blobxfer or S3 compatible services like S3, Minio, Wasabi |
FILESYSTEM |
MODE |
AUTO mode to use internal scheduling routines or MANUAL to simply use this as manual backups only executed by your own means |
AUTO |
MANUAL_RUN_FOREVER |
TRUE or FALSE if you wish to try to make the container exit after the backup |
TRUE |
TEMP_LOCATION |
Perform Backups and Compression in this temporary directory | /tmp/backups/ |
DEBUG_MODE |
If set to true, print copious shell script messages to the container log. Otherwise only basic messages are printed. |
FALSE |
CREATE_LATEST_SYMLINK |
Create a symbolic link pointing to last backup in this format: latest-(DB_TYPE)-(DB_NAME)-(DB_HOST) |
TRUE |
PRE_SCRIPT |
Fill this variable in with a command to execute pre backing up | |
POST_SCRIPT |
Fill this variable in with a command to execute post backing up | |
SPLIT_DB |
For each backup, create a new archive. TRUE or FALSE (MySQL and Postgresql Only) |
TRUE |
| Parameter | Description | Default | _FILE |
|---|---|---|---|
DB_AUTH |
(Mongo/PGSQL Only - Optional) Authentication Database | ||
DB_TYPE |
Type of DB Server to backup couch influx mysql mssql pgsql mongo redis sqlite3 |
||
DB_HOST |
Server Hostname e.g. mariadb. For sqlite3, full path to DB file e.g. /backup/db.sqlite3 |
x | |
DB_NAME |
Schema Name e.g. database or ALL to backup all databases the user has access to. Backup multiple by separating with commas eg db1,db2 |
x | |
DB_NAME_EXCLUDE |
If using ALL - use this as to exclude databases separated via commas from being backed up |
x | |
DB_USER |
username for the database(s) - Can use root for MySQL |
x | |
DB_PASS |
(optional if DB doesn't require it) password for the database | x | |
DB_PORT |
(optional) Set port to connect to DB_HOST. Defaults are provided | varies | x |
INFLUX_VERSION |
What Version of Influx are you backing up from 1.x or 2 series - AMD64 and ARM64 only for 2 |
||
MONGO_CUSTOM_URI |
If you wish to override the MongoDB Connection string enter it here e.g. mongodb+srv://username:password@cluster.id.mongodb.net |
x | |
This environment variable will be parsed and populate the DB_NAME and DB_HOST variables to properly build your backup filenames. You can override them by making your own entries |
Your Organization will be mapped to DB_USER and your root token will need to be mapped to DB_PASS. You may use DB_NAME=ALL to backup the entire set of databases. For DB_HOST use syntax of http(s)://db-name
| Parameter | Description | Default |
|---|---|---|
DB_DUMP_FREQ |
How often to do a dump, in minutes after the first backup. Defaults to 1440 minutes, or once per day. | 1440 |
DB_DUMP_BEGIN |
What time to do the first dump. Defaults to immediate. Must be in one of two formats | |
Absolute HHMM, e.g. 2330 or 0415 |
||
Relative +MM, i.e. how many minutes after starting the container, e.g. +0 (immediate), +10 (in 10 minutes), or +90 in an hour and a half |
||
DB_DUMP_TARGET |
Directory where the database dumps are kept. | ${DB_DUMP_TARGET}/archive/ |
DB_DUMP_TARGET_ARCHIVE |
Optional Directory where the database dumps archives are kept. | |
DB_CLEANUP_TIME |
Value in minutes to delete old backups (only fired when dump frequency fires). 1440 would delete anything above 1 day old. You don't need to set this variable if you want to hold onto everything. | FALSE |
DB_ARCHIVE_TIME |
Value in minutes to move all files files older than (x) from DB_DUMP_TARGET to DB_DUMP_TARGET_ARCHIVE - which is useful when pairing against an external backup system. |
- You may need to wrap your
DB_DUMP_BEGINvalue in quotes for it to properly parse. There have been reports of backups that start with a0get converted into a different format which will not allow the timer to start at the correct time.
| Parameter | Description | Default | _FILE |
|---|---|---|---|
COMPRESSION |
Use either Gzip GZ, Bzip2 BZ, XZip XZ, ZSTD ZSTD or none NONE |
ZSTD |
|
COMPRESSION_LEVEL |
Numerical value of what level of compression to use, most allow 1 to 9 except for ZSTD which allows for 1 to 19 - |
3 |
|
ENABLE_PARALLEL_COMPRESSION |
Use multiple cores when compressing backups TRUE or FALSE |
TRUE |
|
PARALLEL_COMPRESSION_THREADS |
Maximum amount of threads to use when compressing - Integer value e.g. 8 |
autodetected |
|
GZ_RSYNCABLE |
Use --rsyncable (gzip only) for faster rsync transfers and incremental backup deduplication. e.g. TRUE |
FALSE |
|
ENABLE_CHECKSUM |
Generate either a MD5 or SHA1 in Directory, TRUE or FALSE |
TRUE |
|
CHECKSUM |
Either MD5 or SHA1 |
MD5 |
|
EXTRA_OPTS |
If you need to pass extra arguments to the backup and database enumeration command, add them here e.g. --extra-command |
||
EXTRA_DUMP_OPTS |
If you need to pass extra arguments to the backup command only, add them here e.g. --extra-command |
||
EXTRA_ENUMERATION_OPTS |
If you need to pass extra arguments to the database enumeration command only, add them here e.g. --extra-command |
||
MYSQL_MAX_ALLOWED_PACKET |
Max allowed packet if backing up MySQL / MariaDB | 512M |
|
MYSQL_SINGLE_TRANSACTION |
Backup in a single transaction with MySQL / MariaDB | TRUE |
|
MYSQL_STORED_PROCEDURES |
Backup stored procedures with MySQL / MariaDB | TRUE |
|
MYSQL_ENABLE_TLS |
Enable TLS functionality for MySQL client | FALSE |
|
MYSQL_TLS_VERIFY |
(optional) If using TLS (by means of MYSQL_TLS_* variables) verify remote host | FALSE |
|
MYSQL_TLS_VERSION |
What TLS v1.1 v1.2 v1.3 version to utilize |
TLSv1.1,TLSv1.2,TLSv1.3 |
|
MYSQL_TLS_CA_FILE |
Filename to load custom CA certificate for connecting via TLS | /etc/ssl/cert.pem |
x |
MYSQL_TLS_CERT_FILE |
Filename to load client certificate for connecting via TLS | x | |
MYSQL_TLS_KEY_FILE |
Filename to load client key for connecting via TLS | x |
- When using compression with MongoDB, only
GZcompression is possible.
If BACKUP_LOCATION = S3 then the following options are used.
| Parameter | Description | Default | _FILE |
|---|---|---|---|
S3_BUCKET |
S3 Bucket name e.g. mybucket |
x | |
S3_KEY_ID |
S3 Key ID (Optional) | x | |
S3_KEY_SECRET |
S3 Key Secret (Optional) | x | |
S3_PATH |
S3 Pathname to save to (must NOT end in a trailing slash e.g. 'backup') |
x | |
S3_REGION |
Define region in which bucket is defined. Example: ap-northeast-2 |
x | |
S3_HOST |
Hostname (and port) of S3-compatible service, e.g. minio:8080. Defaults to AWS. |
x | |
S3_PROTOCOL |
Protocol to connect to S3_HOST. Either http or https. Defaults to https. |
https |
x |
S3_EXTRA_OPTS |
Add any extra options to the end of the aws-cli process execution |
x | |
S3_CERT_CA_FILE |
Map a volume and point to your custom CA Bundle for verification e.g. /certs/bundle.pem |
x | |
| OR | |||
S3_CERT_SKIP_VERIFY |
Skip verifying self signed certificates when connecting | TRUE |
- When
S3_KEY_IDand/orS3_KEY_SECRETis not set, will try to use IAM role assigned (if any) for uploading the backup files to S3 bucket.
Support to upload backup files with blobxfer to the Azure file share storage.
If BACKUP_LOCATION = blobxfer then the following options are used.
| Parameter | Description | Default | _FILE |
|---|---|---|---|
BLOBXFER_STORAGE_ACCOUNT |
Microsoft Azure Cloud storage account name. | x | |
BLOBXFER_STORAGE_ACCOUNT_KEY |
Microsoft Azure Cloud storage account key. | x | |
BLOBXFER_REMOTE_PATH |
Remote Azure path | /docker-db-backup |
x |
This service uploads files from backup targed directory
DB_DUMP_TARGET. If the a cleanup configuration inDB_CLEANUP_TIMEis defined, the remote directory on Azure storage will also be cleaned automatically.
For debugging and maintenance purposes you may want access the containers shell.
bash docker exec -it (whatever your container name is) bash
Manual Backups can be performed by entering the container and typing backup-now
- Recently there was a request to have the container work with Kubernetes cron scheduling. This can theoretically be accomplished by setting the container
MODE=MANUALand then settingMANUAL_RUN_FOREVER=FALSE- You would also want to disable a few features from the upstream base images specificallyCONTAINER_ENABLE_SCHEDULINGandCONTAINER_ENABLE_MONITORING. This should allow the container to start, execute a backup by executing and then exit cleanly. An alternative way to running the script is to execute/etc/services.available/10-db-backup/run.
Entering in the container and executing restore will execute a menu based script to restore your backups - MariaDB, Postgres, and Mongo supported.
You will be presented with a series of menus allowing you to choose:
- What file to restore
- What type of DB Backup
- What Host to restore to
- What Database Name to restore to
- What Database User to use
- What Database Password to use
- What Database Port to use
The image will try to do auto detection based on the filename for the type, hostname, and database name. The image will also allow you to use environment variables or Docker secrets used to backup the images
The script can also be executed skipping the interactive mode by using the following syntax/
`restore <filename> <db_type> <db_hostname> <db_name> <db_user> <db_pass> <db_port>`
If you only enter some of the arguments you will be prompted to fill them in.
| Parameter | Description | Default |
|---|---|---|
SCRIPT_LOCATION_PRE |
Location on filesystem inside container to execute bash scripts pre backup | /assets/scripts/pre/ |
SCRIPT_LOCATION_POST |
Location on filesystem inside container to execute bash scripts post backup | /assets/scripts/post/ |
If you want to execute a custom script before a backup starts, you can drop bash scripts with the extension of .sh in the location defined in SCRIPT_LOCATION_PRE. See the following example to utilize:
$ cat pre-script.sh
##!/bin/bash
# #### Example Pre Script
# #### $1=DB_TYPE (Type of Backup)
# #### $2=DB_HOST (Backup Host)
# #### $3=DB_NAME (Name of Database backed up
# #### $4=BACKUP START TIME (Seconds since Epoch)ff
# #### $5=BACKUP FILENAME (Filename)
echo "${1} Backup Starting on ${2} for ${3} at ${4}. Filename: ${5}"## script DB_TYPE DB_HOST DB_NAME STARTEPOCH BACKUP_FILENAME
${f} "${dbtype}" "${dbhost}" "${dbname}" "${backup_start_time}" "${target}"
Outputs the following on the console:
`mysql Backup Starting on example-db for example at 1647370800. Filename: mysql_example_example-db_202200315-000000.sql.bz2
If you want to execute a custom script at the end of a backup, you can drop bash scripts with the extension of .sh in the location defined in SCRIPT_LOCATION_POST. Also to support legacy users /assets/custom-scripts is also scanned and executed.See the following example to utilize:
$ cat post-script.sh
##!/bin/bash
# #### Example Post Script
# #### $1=EXIT_CODE (After running backup routine)
# #### $2=DB_TYPE (Type of Backup)
# #### $3=DB_HOST (Backup Host)
# #### #4=DB_NAME (Name of Database backed up
# #### $5=BACKUP START TIME (Seconds since Epoch)
# #### $6=BACKUP FINISH TIME (Seconds since Epoch)
# #### $7=BACKUP TOTAL TIME (Seconds between Start and Finish)
# #### $8=BACKUP FILENAME (Filename)
# #### $9=BACKUP FILESIZE
# #### $10=HASH (If CHECKSUM enabled)
# #### $11=MOVE_EXIT_CODE
echo "${1} ${2} Backup Completed on ${3} for ${4} on ${5} ending ${6} for a duration of ${7} seconds. Filename: ${8} Size: ${9} bytes MD5: ${10}" ## script EXIT_CODE DB_TYPE DB_HOST DB_NAME STARTEPOCH FINISHEPOCH DURATIONEPOCH BACKUP_FILENAME FILESIZE CHECKSUMVALUE
${f} "${exit_code}" "${dbtype}" "${dbhost}" "${dbname}" "${backup_start_timme}" "${backup_finish_time}" "${backup_total_time}" "${target}" "${FILESIZE}" "${checksum_value}" "${move_exit_code}
Outputs the following on the console:
0 mysql Backup Completed on example-db for example on 1647370800 ending 1647370920 for a duration of 120 seconds. Filename: mysql_example_example-db_202200315-000000.sql.bz2 Size: 7795 bytes Hash: 952fbaafa30437494fdf3989a662cd40 0
If you wish to change the size value from bytes to megabytes set environment variable SIZE_VALUE=megabytes
You must make your scripts executable otherwise there is an internal check that will skip trying to run it otherwise.
If for some reason your filesystem or host is not detecting it right, use the environment variable POST_SCRIPT_SKIP_X_VERIFY=TRUE to bypass.
These images were built to serve a specific need in a production environment and gradually have had more functionality added based on requests from the community.
- The Discussions board is a great place for working with the community on tips and tricks of using this image.
- Consider sponsoring me for personalized support
- Please, submit a Bug Report if something isn't working as expected. I'll do my best to issue a fix in short order.
- Feel free to submit a feature request, however there is no guarantee that it will be added, or at what timeline.
- Consider sponsoring me regarding development of features.
- Best effort to track upstream changes, More priority if I am actively using the image in a production environment.
- Consider sponsoring me for up to date releases.
MIT. See LICENSE for more details.