Bareos

1. Overview
2. Common setup
3. Server
4. Client
   1. Client parameters
   2. Jobs
   3. Job presets
      1. mysqldumpbackup
      2. mylvmbackup
      3. percona
      4. pgdumpbackup
      5. s3
      6. Writing your own
   4. Filesets
   5. Complex examples
      1. Pre- and post jobs
      2. Service address

Overview

This module is intended to supplement the configuration of a Bareos (or Bacula) server with the "dynamic" bits, e.g., the definitions of clients, jobs and filesets. The basic configuration (database settings, logging etc.) must be managed in a handwritten bareos-dir.conf. This file should include the configuration generated by Puppet using the following snippet:

# Puppet clients and jobs
@|"bash -c 'ls -d /etc/bareos/{clients,jobs,filesets}.d/*.conf | sed s/^/@/'"

Non-module

Some pieces of the configuration must be written by hand or handled otherwise. This includes schedules and jobdefs.

Common setup

In order to get a working setup, a few common Hiera keys must be set:

bareos::secret: A random string which is hashed with a seed value for each client to create its password.

bareos::director: The name of the director. Not necessarily a hostname. It is essentially the director's username when authenticating with a FD. It is also used in the tag of exported resources. Default: "dump-dir"

bareos::schedules: A hash containing sets of schedules. Each key defines a set, the value for that key is an array of schedule names. A declared job will look for the set called normal by default. The schedules themselves must be defined outside this module.

Optional configuration

bareos::default_jobdef: Name of default job definition. Defaults to DefaultJob. The jobdef itself must be defined outside this module.

bareos::security_zone: A tag identifying which secret the director should use to generate the password for this client.

When security zone is unset, bareos::secret will be used on both client and server. If security zone is set, the client will still use the bareos::secret as seen in its view of Hiera, but the server will instead look for the alternate secret in bareos::server::secrets.

Server

The bareos::server class installs the software and collects exported resources associated with it.

Server parameters

bareos::server::secrets: Hash containing secrets for other security zones. Default: {}

bareos::server::client_file_prefix: Where to put collected client resources. Default: '/etc/bareos/clients.d/'

bareos::server::job_file_prefix: Where to put collected job resources. Default: '/etc/bareos/jobs.d/'

bareos::server::fileset_file_prefix: Where to put collected fileset resources. Default: '/etc/bareos/filesets.d/'

Usage without Hiera

The values bareos::schedules, bareos::secret, bareos::server::secrets are taken directly from Hiera. Users without Hiera can pass the parameters $default_schedules and $default_secret to class bareos and the parameter $default_secrets to class bareos::server.

Client

The bareos::client class will install the file daemon and configure it with passwords. It will also export resources (Client, Job, and Fileset) which the director collects.

Client parameters

bareos::client::password: Set this parameter to get the same password on several clients. This is not the actual password used in the configuration files, this is just a seed which is hashed with ${bareos::secret}. We do that extra step to avoid putting the actual password in PuppetDB. Default: FQDN

bareos::client::concurrency: How many jobs the director will schedule at the same time on this client. The FD will be configured to allow 10 more connections than this. Default: 10

bareos::client::implementation: Either bacula or bareos. This needs to be set via Hiera to affect defaults which are based on this value. Default: "bacula"

bareos::client::client_name: The name of the client, without the "-fd" suffix. Default: FQDN

bareos::client::name_suffix: The suffix to use. Default: "-fd"

bareos::client::address: The address or hostname the director should connect to. Default: FQDN

bareos::client::passive: Turn on passive mode, which makes SD connect to FD rather than the other way round. Requires Bareos 14 and above. Default: false

bareos::client::client_initiated_connection: Turn on Client Initiated Connection, which makes FD connect to DIR on startup and leave the connection up. This enables the FD to have a variable IP address. Requires Bareos 16 and above. Default: false

bareos::client::job_retention: How long to keep jobs from this client in the database. Default: "180d"

bareos::client::file_retention: How long to keep detailed information about backup job contents in the database. Default: "60d"

bareos::client::monitors: Additional list of monitors to add to bacula-fd.conf. Typical use:

bareos::client::monitors:
  tray-mon:
    password: password-in-plain-text

Use eyaml to protect "password-in-plain-text". All keys in the hash are added as parameters to the Director directive.

bareos::client::compression: This variable is only used as a default for filesets declared on this host.

bareos::client::fstype: This variable is only used as a default for filesets declared on this host.

bareos::client::exclude_paths: This variable is only used as a default for filesets declared on this host.

bareos::client::exclude_patterns: This variable is only used as a default for filesets declared on this host.

bareos::client::backup_dir: The default parent directory where the preset jobs will dump data. Default: "/var/backups"

bareos::client::manage_backup_dir: Create and manage the default parent directory. Default: true

bareos::client::backup_dir_owner: Owner of above directory Default: "root"

bareos::client::backup_dir_group: Group of above directory Default: "root"

bareos::client::backup_dir_mode: Mode of above directory Default: "0755"

bareos::client::systemd_limits: Hash of resource limits which needs overriding. Only works for systemd, but no check is done to see if systemd manages the service. Example: { 'nofile' => 16384 }. Default is to do nothing.

bareos::client::ipv6: Whether to listen on IPv6 when available. This is by default set to false on Windows due to a bug in Bareos.

bareos::client::tls_enable: Whether to enable TLS. Set to true or false, default is to omit directive. (Proper TLS support will come in a later version of this module.)

bareos::client::package: The name of the package(s) to install. The default is chosen based on operatingsystem and implementation. On Windows this is set to an empty array since the package name contains the version number, and Puppet usually will not be able to install the package. In other words, the Bareos client must be installed before Puppet is run.

In addition, you can manage the service name, the port number, the location of the log file, the pid file and the working directory, but this should not be necessary to do.

Jobs

Jobs are defined in the bareos::client::jobs hash.

By default, a "system" job will be enabled for the client, using the default jobdef. If you define a "system" fileset in the bareos::client::filesets hash, it will be used with this default job. (It will not work for filesets declared directly with the bareos::client::fileset define.)

Each key in the hash becomes the name of the resource. This is added to the client name and used as the name of the job. A bareos::job_definition with that name will be exported for the director to pick up.

job_name: Specify the full job name explicitly.

jobdef: The name of the job defaults. Default: $bareos::default_jobdef

fileset: The name of the fileset to use. When set, overrides the fileset defined in the jobdef. This can be the full name of the fileset, but also the abbreviated name used in bareos::client::filesets. If unset, it will look for short name of this job (e.g., "system"), and use that if it exists.

schedule_set: The name of the list of schedules to pick randomly from. Default: normal

sched: Explicit name of schedule, overrides random selection. (schedule is a reserved word in Puppet, hence the strange parameter name.)

accurate: Whether to turn Accurate on or off. When set to '', don't include directive in configuration (job defaults are used). Default: ''

order: Give a hint to in what order jobs should run. This should be a value consisting of one capital letter and two digits. Jobs with "A00" will be scheduled first, "Z99" will be scheduled last. The default is "N50".

runscript: Array of script specifications to run before or after job. Each element is a hash containing values for command and optional other parameters. command can be a single command or an array of strings. runswhen is either before or after, by default it is before. Other parameters are written verbatim as Key = value to bareos configuration.

preset: Use specified define to export the job. See next section. If unqualified, bareos::job::preset:: will be prepended.

preset_params: Parameters to pass to preset class.

base_job_name: Optional name of a Base backup job. If Bareos has a valid copy of this Base job, a Full backup will be relative to that base.

make_base_job: Boolean which controls whether a companion Base level job should be defined. The base backup job definition will be the same as this job, but with a different schedule and jobdefs. Default is false.

base_schedule: What schedule set to use for the Base job. Must have the same number of entries as the schedule set for the job itself. Default is 'base'.

base_jobdef: The jobdefs to use for the Base job. Default is 'BaseJob'.

Usage example

The following example installs a backup agent and registers a job with all the default settings:

include bareos::client

To only install the client without running any jobs, add to Hiera:

bareos::client::jobs: {}

This example also runs a normal full backup, but later than normal (this assumes that a set called late is defined in bareos::schedules):

bareos::client::jobs:
  system:
    schedule_set: late

Job presets

A preset define can install scripts or other software on the client in addition to exporting configuration for the backup server.

mysqldumpbackup

This preset installs the script mysqldumpbackup and installs a configuration file for it. See code for full list of parameters.

Example usage:

bareos::client::jobs:
  system:
    preset: bareos::job::preset::mysqldumpbackup
    preset_params:
      keep_backup: 5
      backup_dir: /srv/mysql/backup

If using MariaDB, add this to the preset_params block:

      servicename: mariadb

mylvmbackup

This preset installs the package mylvmbackup and installs a configuration file for it. See code for full list of parameters.

Example usage:

bareos::client::jobs:
  system:
    preset: bareos::job::preset::mylvmbackup
    preset_params:
      keep_backup: 5
      vgname: sysvg
      lvname: mysql

percona

This preset installs the package percona-xtrabackup and the percona plugin from bareos-contrib. (Since the plugin is not packaged, we distribute a copy of it in this module.) The backup will include a "virtual" file in xbstream format and the binary logs. You should still include a normal system backup.

Available settings in preset_params include:

mycnf: location of my.cnf to use

skip_binlog: do not include binlogs in backup, default is false

xtrabackup_package: name of package containing xtrabackup(1). On Ubuntu Xenial you may need to specify "percona-xtrabackup-24". Default: "percona-xtrabackup"

Example usage:

bareos::client::implementation: bareos
bareos::client::jobs:
  system: {}
  db:
    preset: bareos::job::preset::percona
    preset_params:
      mycnf: /etc/mysql/mysql-db02.cnf

pgdumpbackup

This preset installs the script pgdumpbackup and installs a configuration file for it. It has the following params, passed via preset_params:

instance: name of instance. Needed if there is more than one job using this preset. Will be added as argument to pgdumpbackup unless it is default.

ignore_not_running: if true, exit silently without taking backup if postgresql is not running. Default is false.

keep_backup: how many days to keep backup

backup_dir: where to store backups

backup_dir_owner: what user should own backup_dir. Default: 'postgres'

backup_dir_group: what group should own backup_dir. Default: 'root'

backup_dir_mode: what permissions to use on backup_dir. Default: '0750'

server: server name to connect to (default is local socket)

initscript: to check if service is running

cluster: what cluster to dump (default "", which means connect to port 5432)

skip_databases: array of databases to skip

log_method: where to log. default is "console" (ie., stderr)

syslog_facility: where to log. default is 'daemon'

environ: array of extra environment variables (example: ["HOME=/root"])

Example usage:

bareos::client::jobs:
  system:
    preset: bareos::job::preset::pgdumpbackup
    preset_params:
      cluster: 9.2/main

s3

This preset installs an S3 plugin for Bareos (Bacula is not supported). (Since the plugin is not packaged, we distribute a copy of it in this module.) The backup will include a "virtual" file directory tree /situla with a traversed copy of the bucket inside. A script which calls radosgw-admin to create an authentication file is also included.

Available settings in preset_params include:

user_name: who to authenticate as

bucket: what bucket to backup.

prefix: an optional prefix to restrict the backup

pattern: an optional regexp to restrict the backup to matching files.

For convenience, the job name can be used to specify these values, if it is on the format anything:user_name:bucket:anything. bucket is optional, if it is unspecified, it is assumed to be the same as the user name.

Example usage:

bareos::client::implementation: bareos
bareos::client::jobs:
  # backup of bucket websrv as user websrv
  "s3:websrv":
    preset: bareos::job::preset::s3
    preset_params:
      pattern: "\.pdf$"
  # backup of bucket aux as user websrv
  "s3:websrv:aux":
    preset: bareos::job::preset::s3
  # backup of bucket aux2, subdirectory img, as user websrv
  "s3:websrv:aux2:img":
    preset: bareos::job::preset::s3
    preset_params:
      prefix: img
  # backup of bucket aux2, subdirectory docs, as user websrv
  "docs":
    preset: bareos::job::preset::s3
    preset_params:
      user_name: websrv
      bucket: aux2
      prefix: docs

Writing your own preset

The signature for a preset should be this:

define widget::backup::preset::widgetdump(
    $short_title,
    $client_name,
    $jobdef,
    $fileset,
    $sched,
    $order,
    $runscript,
    $params,
)

title for the define will be the full job name.

short_title will be the short job name, for your convenience.

client_name is the name of the client, and should be passed on.

jobdef will be the empty string if the user didn't specify a jobdef explicitly. You should respect the explicit value, but replace the empty string with a value which works for your preset (possibly $bareos::default_jobdef). New job defaults can not be defined in Puppet code, they must be added to the main Bareos configuration manually.

fileset will normally be empty, and should just be passed on.

sched is the chosen schedule for this job, pass it on.

order should just be passed on.

params is the hash passed by the user as preset_params. The preset is free to specify its format and content.

runscript is an array of additional pre- or post commands.

A normal job exports a bareos::job_define which bareos::server picks up. When a preset is used, exporting that declaration must be done by its define.

This should be done like this:

$_jobdef = $jobdef ? { '' => 'WidgetJob', default => $jobdef }
@@bareos::job_definition {
  $title:
    client_name => $client_name,
    name_suffix => $bareos::client::name_suffix,
    base        => $base,
    jobdef      => $_jobdef,
    fileset     => $fileset,
    sched       => $sched,
    order       => $order,
    runscript   => flatten([ $runscript,
                             [{ 'command' => '/usr/local/bin/widgetdump' }]
                           ]),
    tag         => "bareos::server::${bareos::director}"
}

Almost all of the above code must be copied more or less verbatim. flatten() is used to concatenate arrays, which is otherwise only available when the future parser is enabled. If you don't need a runscript, you just pass $runscript on.

You should try to write your define so it can be used more than once per client, i.e., consider using ensure_resource('file', { ... }) instead of just file to avoid duplicate declarations.

Filesets

The support for filesets is not complete, it is kept simple to focus on filesets with static include and exclude rules.

Since the normal filesets should contain Exclude dir containing .nobackup, an alternative to making a custom fileset may be to manage strategically placed .nobackup files using Puppet.

The name (title) of the bareos::client::fileset instance is added to the client name and used as the name of the fileset in the Bareos configuration. The define exports a fileset_definition which the director will pick up.

fileset_name: Specify the fileset name explicitly.

client_name: Used as the first part of a fully qualified fileset name. Doesn't need to relate to any host or client names as long as the result is unique. Default is bareos::client::client_name.

include_paths: Array of paths to include in backup. Mandatory, no default.

exclude_paths: Array of paths to exclude. If unset, use the client's default (bareos::client::exclude_paths). If one of the values is "defaults", the default exclude list will be added to the array.

exclude_patterns: Hash of different kinds of patterns to exclude. If unset, use the client's default (bareos::client::exclude_patterns). The possible keys are wild_dir, wild_file, regex_dir and regex_file. Each key in the hash can have a list of patterns as its value.

exclude_dir_containing: Directories containing a file with this name will be skipped. Set to "" to disable functionality. Default: ".nobackup".

include_patterns: Hash of different kinds of patterns to include. Default is empty (no special rules). The possible keys are wild_dir, wild_file, regex_dir and regex_file. Each key in the hash can have a list of patterns as its value. In Bareos, all files will be included by default, so specifying an extra include rule changes nothing. Therefore, an exclude rule matching all files will be added to the configuration as well. It is recommended to manually inspect the result on the server to see if it works as intended. This is especially important for the wild_dir and regex_dir directives.

ignore_changes: If fileset changes, rerun Full backup if this is set to false. Default: true

acl_support: Include information about ACLs in backup. Causes an extra system call per file. Default: true

compression: What compression algorithm to use. To disable compression, set to false. Default: "GZIP"

sparse: Whether to store information about the holes in sparse files or just store the holes as plain NUL bytes. Default: true

onefs: Whether to recurse into mount points. Default: false (do not recurse).

fstype: If onefs is false (the default), this array lists the filesystem types which should be recursed into (and backed up). The default (from bareos::client::fstype) contains the normal local filesystems, like ext4 and xfs, but not nfs or cifs.

Fileset example

bareos::client::filesets:
  only_srv:
    include_paths:
      - /srv
    exclude_paths:
      - /srv/cache

If this configuration is used on node foo.example.com, a fileset called "foo.example.com-only_srv" will be exported.

Complex examples

Pre- and post jobs

The following will install the software and register two jobs, "system" and "srv". Before the "srv" job runs, the prepare script will run, and if it fails, backup will be aborted. After the "srv" job finishes, the cleanup script will run.

bareos::client::jobs:
  system:
    fileset: "system"
  srv:
    fileset: "srv"
    runscript:
      - command: "/usr/local/sbin/prepare"
        abortjobonerror: true
      - command: "/usr/local/sbin/cleanup"
        runswhen: after

bareos::client::filesets:
  system:
    include_paths:
      - /
    exclude_paths:
      - /srv
  srv:
    include_paths:
      - /srv

NFS

The default fileset will not traverse into NFS file systems, so it needs to be specified explicitly. Here we define a separate job for NFS paths. We set OneFS to true, since the default fstype list does not include nfs. Alternatively, if we want to recurse into other NFS filesystems, we could set fstype: ["nfs"]

bareos::client::filesets:
  nfs:
    onefs: true
    include_paths:
      - /srv/data

We add another job which uses the new fileset (the system job still uses the default fileset specified in the $bareos::default_jobdef)

bareos::client::jobs:
  system: {}
  nfs:
    fileset: "nfs"

Service address

When doing backup of a service which can run on more than one node, it is essential to set bareos::client::password to the same value on each of the nodes. It is also important that the job and fileset definitions agree across nodes, so it is best to put the configuration in a single Hiera file read by the relevant nodes.

Example:

bareos::client::password: common-secret

bareos::client::jobs:
  system:
    fileset: without_archive
  archive_service:
    client_name: archive.example.com
    fileset_name: archive

bareos::client::filesets:
  without_archive:
    client_name: archive.example.com
    include_paths:
      - /
    exclude_paths:
      - /srv/archive
  archive:
    client_name: archive.example.com
    include_paths:
      - /srv/archive

bareos::client::service_addr:
  archive.example.com: {}

This will make one job "${fqdn}-system-job" for each node using this configuration, and one job "archive.example.com-archive_service-job". The latter job will run on the extra client, "archive.example.com". (The name must resolve in DNS, otherwise specify "address".)

It will declare one fileset, "archive.example.com-without_archive", which all the "${fqdn}-system-job" jobs use. If the fileset didn't specify "client_name", each node would declare its own copy of the fileset called "${fqdn}-without_archive". This would work fine, but it is always good to avoid duplication.

Finally, it will declare one fileset "archive.example.com-archive", which the "archive.example.com-archive_service-job" uses.