Skip to content

Commit

Permalink
Merge pull request #738 from NethServer/restic-wrapper
Browse files Browse the repository at this point in the history
Add Restic wrapper helper command

Refs NethServer/dev#7072
  • Loading branch information
DavidePrincipi authored Oct 23, 2024
2 parents 4a00569 + cb3d138 commit b3608c7
Show file tree
Hide file tree
Showing 3 changed files with 162 additions and 12 deletions.
116 changes: 116 additions & 0 deletions core/imageroot/usr/local/agent/bin/restic-wrapper
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
#!/usr/bin/env python3

#
# Copyright (C) 2024 Nethesis S.r.l.
# SPDX-License-Identifier: GPL-3.0-or-later
#


import agent
import json
import sys
import os, os.path
import argparse

DESCRIPTION="""
Run Restic within a Podman container, using the repository configuration
from NethServer 8 modules.
Options specific to the wrapper are supported, with any additional options
passed directly to Restic.
When executed in the agent environment of a specific module, the minimum
backup ID is assumed by default. To use a different backup schedule,
specify it with "--backup". Default module volumes are automatically
mounted.
In the cluster agent environment, specify the Restic repository manually
using "--destination" and "--repopath". Volumes must be manually mounted
by passing additional options to Podman via "--podman".
Example invocations:
restic-wrapper --show
restic-wrapper stats
restic-wrapper --backup 2 stats
restic-wrapper --destination fd4e75b5-9836-42be-b5db-b4983a851e40 --repopath webtop/2f3c56e2-aa1a-4eda-9e32-0ec9348cd31b stats
restic-wrapper --podman volume=./restoredir:/srv/restoredir restore latest:state/ -t /srv/restoredir
"""

parser = argparse.ArgumentParser(
prog="restic-wrapper",
formatter_class=argparse.RawDescriptionHelpFormatter,
description=DESCRIPTION,
add_help=False, # Help flag is also forwarded to Restic
)
parser.add_argument('--podman', action="append", type=str, help="Arguments for Podman, e.g. --podman volume=./mydir:/srv/path")
parser.add_argument('--backup', help="Backup ID")
parser.add_argument('--destination', help="Destination UUID")
parser.add_argument('--repopath', help="Repository path under the backup destination")
parser.add_argument('--show', action="store_true", help="Show destinations and backups")
wargs, rargs = parser.parse_known_args()
if '--help' in rargs:
# Print the wrapper helper, then continue, to forward the help flag to Restic
parser.print_help()

rdb = agent.redis_connect(use_replica=True) # Connect to local replica

if wargs.show:
header="No backup destination found.\n"
listing=""
for krepo in rdb.scan_iter('cluster/backup_repository/*'):
header="Destinations:\n"
adest = rdb.hgetall(krepo)
listing += f"- {krepo.removeprefix('cluster/backup_repository/')} {adest['name']} ({adest['url']})\n"
print(header + listing, end="")
header="No backup schedule found.\n"
listing=""
for kbackup in rdb.scan_iter('cluster/backup/*'):
header="Scheduled backups:\n"
bid = kbackup.removeprefix('cluster/backup/')
abackup = rdb.hgetall(kbackup)
listing += f"- {bid} {abackup['name']}, destination UUID {abackup['repository']}\n"
print(header + listing, end="")
sys.exit(0)

module_id = os.environ.get("MODULE_ID")
repopath = None
backup_id = None
dest_uuid = None
podman_args = ["--workdir=/srv"]
if wargs.podman:
podman_args.extend(['--' + arg for arg in wargs.podman])
if wargs.backup:
backup_id = wargs.backup
if module_id:
podman_args.extend(agent.get_state_volume_args()) # get volumes from state-include.conf
repopath = agent.get_image_name_from_url(os.environ["IMAGE_URL"]) + "/" + os.environ["MODULE_UUID"]
if not backup_id:
try:
backup_id = min(rdb.smembers(f"module/{module_id}/backups"))
except:
pass

if backup_id:
dest_uuid = rdb.hget(f"cluster/backup/{backup_id}", "repository")
else:
dest_uuid = wargs.destination

if not dest_uuid:
parser.print_help()
print()
print(f"Destination not found. Consider using --backup or --destination arguments.", file=sys.stderr)
sys.exit(2)

if wargs.repopath:
repopath = wargs.repopath

if not repopath:
parser.print_help()
print()
print(f"Path for Restic repository not found. Consider using --repopath argument.", file=sys.stderr)
sys.exit(2)

proc = agent.run_restic(rdb, dest_uuid, repopath, podman_args, rargs)
sys.exit(proc.returncode)
55 changes: 43 additions & 12 deletions docs/core/backup_restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,14 @@ module and restore it to a different node or cluster.

## Design

The backup engine is [restic](https://restic.net/) which runs inside a container along with [rclone](https://rclone.org/)
used to inspect the backup repository contents.
The backup engine is [Restic](https://restic.net/) which runs inside a
container along with [Rclone](https://rclone.org/) used to inspect the
backup repository contents.

Backups are saved inside *backup repositories*, remote network-accessible spaces where data are stored.
No local storage backup is possible (eg. USB disks).
A backup repository can contain multiple backup instances, each module instance has its own sub-directory to avoid conflicts.
Backups are saved inside *backup destinations*, remote spaces where data
are stored. A backup destination can contain multiple backup instances,
each module instance has its own sub-directory to avoid conflicts. This
sub-directory is the root of the module instance Restic repository.

The system implements the common logic for backup inside the agent with `module-backup` command.
Each module can implement `module-dump-state` and `module-cleanup-state` to prepare/cleanup the data that has to be included in the backup.
Expand All @@ -30,17 +32,18 @@ The basic `10restore` step actually runs the Restic restore procedure in a tempo

All backups are scheduled by systemd timers. Given a backup with id `1`, it is possible to retrieve the time status with:
- rootless containers, eg. `dokuwiki1`, executed by `dokuwiki1` user: `systemctl --user status backup1.timer`
- rootfull containers, eg. `samba1`, executed by `root` user: `systemctl status backup1-samba1.timer`
- rootfull containers, eg. `dnsmasq1`, executed by `root` user: `systemctl status backup1-dnsmasq1.timer`

## Include and exclude files

Whenever possible, containers should always use volumes to avoid SELinux issues during backup an restore.
Whenever possible, containers should use volumes to avoid UID/GID
namespace mappings and SELinux issues during backup an restore.

Includes can be added to the `state-include.conf` file saved inside `AGENT_INSTALL_DIR/etc/`.
In the [source tree](modules/images/#source-tree), the file should be placed under `<module>/imageroot/etc/state-include.conf`.
On installed modules, the file will appear on different paths:
- rootless containers, eg. `dokuwiki1`, full path will be `/home/dokuwiki1/.config/etc/state-include.conf`
- rootfull containers, eg. `samba1`, full path will be `/var/lib/nethserver/samba1/etc/state-include.conf`
- rootfull containers, eg. `dnsmasq1`, full path will be `/var/lib/nethserver/dnsmasq1/etc/state-include.conf`

Lines are interpreted as path patterns. Only patterns referring to
volumes and the agent `state/` directory are considered.
Expand All @@ -62,7 +65,7 @@ Internally, volumes will be mapped as:
`dokuwiki-data`

- `<module_id>-<volume_name>` for rootfull containers; eg. for module
`samba1`, line prefix `volumes/ data` maps to volume name `samba1-data`
`dnsmasq1`, line prefix `volumes/ data` maps to volume name `dnsmasq1-data`

Volumes listed in `state-include.conf` are automatically mounted (and
created if necessary) by the basic `10restore` step of the
Expand Down Expand Up @@ -147,9 +150,9 @@ api-cli run add-backup-repository --data '{"name":"BackBlaze repo1","url":"b2:ba
{"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "id": "48ce000a-79b7-5fe6-8558-177fd70c27b4"}
```

3. Create a new daily backup named `mybackup` with a retention of 3 snapshots (3 days) which includes `dokuwiki1` and `samba1` instances:
3. Create a new daily backup named `mybackup` with a retention of 3 snapshots (3 days) which includes `dokuwiki1` and `dnsmasq1` instances:
```
api-cli run add-backup --data '{"repository":"48ce000a-79b7-5fe6-8558-177fd70c27b4","schedule":"daily","retention":3,"instances":["dokuwiki1","samba1"],"enabled":true, "name":"mybackup"}'
api-cli run add-backup --data '{"repository":"48ce000a-79b7-5fe6-8558-177fd70c27b4","schedule":"daily","retention":3,"instances":["dokuwiki1","dnsmasq1"],"enabled":true, "name":"mybackup"}'
```

4. The output will the id of the backup:
Expand All @@ -164,7 +167,7 @@ api-cli run run-backup --data '{"id":1}'

For debugging purposes, you can also launch systemd units:
- rootless container, eg. `dokuwiki1`: `runagent -m dokuwiki1 systemctl --user start backup1.service`
- rootfull container, eg. `samba1`: systemctl start backup1-samba1.service
- rootfull container, eg. `dnsmasq1`: systemctl start backup1-dnsmasq1.service

To remove the backup use:
```
Expand Down Expand Up @@ -203,3 +206,31 @@ To decrypt it run a command like this:

The restore procedure can be started from the UI of a new NS8
installation: upload the file and specify the password from the UI.

## The `restic-wrapper` command

The Restic binary is not installed in the host system. NS8 runs Restic
within a core container, preparing environment variables with values read
from the Redis DB and properly mounting the application Podman volumes.

The `restic-wrapper` command is designed to manually run Restic from the
command line. It can help to restore individual files and directories, or
run maintenance commands on remote Restic repositories.

The command can be invoked from any agent environment. Print its inline
help with this command:

runagent restic-wrapper --help

Some options require a module backup ID, or backup destination UUID. Use
the `--show` option to list them. For example:

runagent -m mail1 restic-wrapper --show

Example of output:

Destinations:
- dac5d576-ed63-5c4b-b028-c5e97022b27b OVH S3 destination (s3:s3.de.io.cloud.ovh.net/ns8-backups)
- 14030a59-a4e6-57cc-b8ea-cd5f97fe44c8 BackBlaze repo1 (b2:ns8-backups)
Scheduled backups:
- 1 Backup to BackBlaze repo1, destination UUID 14030a59-a4e6-57cc-b8ea-cd5f97fe44c8
3 changes: 3 additions & 0 deletions docs/modules/certification.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ parent: Modules

# Application Community Certification

* TOC
{:toc}

## From ideas to applications

Before creating an application for NethServer 8, consider if it's really
Expand Down

0 comments on commit b3608c7

Please sign in to comment.