System Configuration Backup

The system backup contains system-wide configuration files, keys, certificates (i.e. SSH keys) and the datra about 3rd-party software package repositories and installed packages. This includes:

The /etc directory;

The /opt directory;

The /usr/local directory;

The /var/lib directory;

A list of currently installed Ubuntu software packages (apt);

A list of currently installed Python packages (pip);

Borg Preparation 

Add configuration and keys directory:

$ sudo mkdir -p /etc/borg/{keys,ssh}

Add cache and security directories:

$ sudo mkdir -p /var/lib/borg/{cache,security}

Create SSH private and public key:

$ sudo ssh-keygen -t ed25519 -C "BorgBackup@$(hostname)" -f /etc/borg/ssh/id_ed25519
$ sudo chmod 0600 /etc/borg/ssh/id_ed25519
$ sudo cat /etc/borg/ssh/id_ed25519.pub

Install the public key on the backup server:

$ sudo ssh-copy-id -i /etc/borg/ssh/id_ed25519.pub borg-backup@nas.example.net

The backup server needs then to setup that public key for use with this specific Borg client by defining a ssh forced command, that points Borg to this clients repository.

Mail Notification Script 

The /etc/borgmatic/notify.sh shell script will send us a mail message, whenever something goes wrong during backups, pruning or a repository check.

#!/usr/bin/bash
#
# Notify user of borgmatic backup error.
#
# /etc/notify.sh "{configuration_filename}" "{repository}" "{error}" "{output}"

mail -s "Borgmatic Error on ${HOST}" "${USER}" <<EOF

Borgmatic backup on ${HOST} failed.

Configuration file: $1

Repository: $2

Error Message: $3

Command output, if any: $4

For more information, query the systemd journal on ${HOST}.

EOF

Borgmatic Configuration 

Generate a new borgmatic configuration file:

$ sudo generate-borgmatic-config

This generates a sample configuration file /etc/borgmatic/config.yaml.

What to Backup and Where 

# Where to look for files to backup, and where to store those backups. See
# https://borgbackup.readthedocs.io/en/stable/quickstart.html and
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create for details.
location:
    # List of source directories to backup (required). Globs and tildes are expanded.
    source_directories:
        - /boot
        - /etc
        - /opt
        - /root
        - /usr/local
        - /var

    # Paths to local or remote repositories (required). Tildes are expanded. Multiple
    # repositories are backed up to in sequence. See ssh_command for SSH options like
    # identity file or port.
    repositories:
        - ssh://borg-backup@nas.example.net/volume1/BorgBackup/{hostname}

    # Any paths matching these patterns are excluded from backups. Globs and tildes
    # are expanded. See the output of "borg help patterns" for more details.
    # Any paths matching these patterns are excluded from backups. Globs and tildes
    # are expanded. See the output of "borg help patterns" for more details.
    exclude_patterns:
        - '**/.cache'
        - '**/cache'
        - /dev
        - /lost+found
        - /mnt
        - /proc
        - /run
        - /snap
        - /sys
        - /var/lib/container
        - /var/lib/lxcfs
        - /var/lib/ooni
        - /var/run
        - /var/tmp/*

    # Exclude directories that contain a CACHEDIR.TAG file. See
    # http://www.brynosaurus.com/cachedir/spec.html for details. Defaults to false.
    exclude_caches: true

    # Exclude directories that contain a file with the given filenames. Defaults to not
    # set.
    exclude_if_present:
        - .nobackup

    # Path for additional source files used for temporary internal state like
    # borgmatic database dumps. Note that changing this path prevents "borgmatic
    # restore" from finding any database dumps created before the change. Defaults
    # to ~/.borgmatic
    borgmatic_source_directory: /tmp/borgmatic

How to Store the Backups 

# Repository storage options. See
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for
# details.
storage:
    # Passphrase to unlock the encryption key with. Only use on repositories that were
    # initialized with passphrase/repokey encryption. Quote the value if it contains
    # punctuation, so it parses correctly. And backslash any quote or backslash
    # literals as well. Defaults to not set.
    encryption_passphrase: "********"

    # Command to use instead of "ssh". This can be used to specify ssh options.
    # Defaults to not set.
    ssh_command: ssh -i /etc/borg/ssh/id_ed25519 -o BatchMode=yes -o VerifyHostKeyDNS=yes

    # Base path used for various Borg directories. Defaults to $HOME, ~$USER, or ~.
    # See https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for details.
    borg_base_directory: /var/lib//borg

    # Path for Borg configuration files. Defaults to $borg_base_directory/.config/borg
    borg_config_directory: /etc/borg

    # Path for Borg cache files. Defaults to $borg_base_directory/.cache/borg
    borg_cache_directory: /var/lib/borg/cache

    # Path for Borg encryption key files. Defaults to $borg_base_directory/.config/borg/keys
    borg_keys_directory: /etc/borg/keys

    # Name of the archive. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Defaults to
    # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must
    # also specify a prefix in the retention section to avoid accidental pruning of
    # archives with a different archive name format. And you should also specify a
    # prefix in the consistency section as well.
    archive_name_format: '{hostname}-{now}'

How Long to Keep Backups 

# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
# At least one of the "keep" options is required for pruning to work. See
# https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/
# if you'd like to skip pruning entirely.
retention:
    # Keep all archives within this time interval.
    keep_within: 6H

    # Number of hourly archives to keep.
    keep_hourly: 8

    # Number of daily archives to keep.
    keep_daily: 7

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

    # Number of yearly archives to keep.
    keep_yearly: 1

    # When pruning, only consider archive names starting with this prefix.
    # Borg placeholders can be used. See the output of "borg help placeholders" for
    # details. Defaults to "{hostname}-". Use an empty value to disable the default.
    prefix: '{hostname}-'

What To Do on Errors 

# Shell commands, scripts, or integrations to execute at various points during a borgmatic
# run. IMPORTANT: All provided commands and scripts are executed with user permissions of
# borgmatic. Do not forget to set secure permissions on this configuration file (chmod
# 0600) as well as on any script called from a hook (chmod 0700) to prevent potential
# shell injection or privilege escalation.
hooks:

    # List of one or more shell commands or scripts to execute when an exception
    # occurs during a "prune", "create", or "check" action or an associated
    # before/after hook.
    # Supported variables:
    #   configuration_filename
    #   repository
    #   error
    #   output
    on_error:
       - /etc/borgmatic/notify.sh "{configuration_filename}" "{repository}" "{error}" "{output}"

    # List of one or more shell commands or scripts to execute before running all

What To Do Before and After 

    # List of one or more shell commands or scripts to execute before running all
    # actions (if one of them is "create"). These are collected from all configuration
    # files and then run once before all of them (prior to all actions).
    # before_everything:
        # - echo "Starting actions."

    # List of one or more shell commands or scripts to execute after running all
    # actions (if one of them is "create"). These are collected from all configuration
    # files and then run once before all of them (prior to all actions).
    # after_everything:
        # - echo "Completed actions."

Secure and Validate Configuration 

To prevent potential shell injection or privilege escalation, do not forget to set secure permissions on borgmatic configuration files (chmod 0600) and scripts (chmod 0700) invoked by hooks.

$ sudo chmod 0600 /etc/borgmatic/config.yml
$ sudo chmod 0700 /etc/borgmatic/notify.sh
$ sudo validate-borgmatic-config

Initialize Repository 

$ sudo borgmatic init --encryption repokey

Interactive Backup Test 

$ sudo borgmatic --verbosity 1 --files

Systemd Service Files 

Service 

Start a system backup once, but only if connected to a network and not running on battery.

Systemd service file: /etc/systemd/system/borgmatic.service:

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true

[Service]
Type=oneshot

# Lower CPU and I/O priority.
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no
# Prevent rate limiting of borgmatic log events. If you are using an older
# version of systemd that doesn't support this (pre-240 or so), you may have to
# remove this option.
LogRateLimitIntervalSec=0

# Delay start to prevent backups running during boot. 
ExecStartPre=sleep 1m

# Don't shutdown, sleep or idle during backups.
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /usr/local/bin/borgmatic --syslog-verbosity 1
# Note that systemd-inhibit requires dbus and dbus-user-session to be installed.

Schedule the system backups once every day at a random time between midnight and 6 AM. or if a scheduled backup time was missed, due to the system powered off or asleep. Don’t wake up the system just for doing a backup.

Systemd timer file /etc/systemd/system/borgmatic.timer:

[Unit]
Description=Run borgmatic system backup

[Timer]
# Schedule the start of the sevice to every day at midnight.
OnCalendar=daily

# Delay the start of the service by a radmon time of up to 6 hours
RandomizedDelaySec=6h

# Only wake up the system for this service after 8 hours or more.
#AccuracySec=8h

# Catch up on missed runs of the service when the system was powered down.
Persistent=true

[Install]
WantedBy=timers.target

Activate 

$ sudo systemctl enable --now borgmatic.timer