User Data Backup

The user data backup basically consists of the users home directory, with some exeptions.

Borg Preparation

Add configuration and keys directory:

$ mkdir -p ~/.config/borg/{keys,ssh,security}
$ chmod -R 0700 ~/.config/borg

Add cache directory:

$ mkdir -p ~/.cache/borg

Create SSH private and public keys for use with Borg:

$ ssh-keygen -t ed25519 -f ~/.config/borg/ssh/id_ed25519
$ chmod 0600 ~/.config/borg/ssh/id_ed25519
$ cat ~/.config/borg/ssh/id_ed25519.pub

Install the public key on the backup server:

$ ssh-copy-id -i ~/.config/borg/ssh/id_ed25519.pub borg-backup@nas.example.net

The backup server needs then to setup that public key for use with this specific Borg client by defining a ssh forced command, that points Borg to this clients repository.

Mail Notification Script

The /.config/borgmatic/notify.sh shell script will send us a mail message, whenever something goes wrong during backups, pruning or a repository check.

#!/usr/bin/bash
#
# Notify user of borgmatic backup error.
#
# /etc/notify.sh "{configuration_filename}" "{repository}" "{error}" "{output}"

mail -s "Borgmatic Error on ${HOST}" "${USER}" <<EOF

Borgmatic backup on ${HOST} failed.

Configuration file: $1

Repository: $2

Error Message: $3

Command output, if any: $4

For more information, query the systemd journal on ${HOST}.

EOF

Borgmatic Configuration

Generate a new borgmatic configuration file:

$ generate-borgmatic-config -d ~/.config/borgmatic/config.yaml

This generates a sample configuration file /home/user/.config/borgmatic/config.yaml.

What to Backup and Where

# Where to look for files to backup, and where to store those backups. See
# https://borgbackup.readthedocs.io/en/stable/quickstart.html and
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create for details.
location:
    # List of source directories to backup (required). Globs and tildes are expanded.
    source_directories:
        - ~/

    # Paths to local or remote repositories (required). Tildes are expanded. Multiple
    # repositories are backed up to in sequence. See ssh_command for SSH options like
    # identity file or port.
    repositories:
        - ssh://borg-backup@nas.example.net/volume1/BorgBackup/{user}-{hostname}

    # Any paths matching these patterns are excluded from backups. Globs and tildes
    # are expanded. See the output of "borg help patterns" for more details.
    exclude_patterns:
        - '**/.cache'
        - '**/cache'        
        - '**/.beagle'
        - ~/.eve/ResFiles
        - ~/.local/share/keybase/fs
        - ~/.local/share/keybase/kbfs_block_cache
        - ~/.local/share/keybase/kbfs_sync_cache
        - ~/.local/share/keybase/keybase.leveldb
        - ~/.local/share/Trash
        - ~/.thumbnails
        - ~/.Trash
        - ~/Downloads
        - ~/Music
        - ~/Videos

    # Exclude directories that contain a CACHEDIR.TAG file. See
    # http://www.brynosaurus.com/cachedir/spec.html for details. Defaults to false.
    exclude_caches: true

    # Exclude directories that contain a file with the given filenames. Defaults to not
    # set.
    exclude_if_present:
        - .nobackup

How to Store the Backups

# Repository storage options. See
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for
# details.
storage:
    # Passphrase to unlock the encryption key with. Only use on repositories that were
    # initialized with passphrase/repokey encryption. Quote the value if it contains
    # punctuation, so it parses correctly. And backslash any quote or backslash
    # literals as well. Defaults to not set.
    encryption_passphrase: "********"

    # Command to use instead of "ssh". This can be used to specify ssh options.
    # Defaults to not set.
    ssh_command: ssh -i ~/.config/borg/ssh/id_ed25519 -o BatchMode=yes -o VerifyHostKeyDNS=yes
                 
    # Name of the archive. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Defaults to
    # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must
    # also specify a prefix in the retention section to avoid accidental pruning of
    # archives with a different archive name format. And you should also specify a
    # prefix in the consistency section as well.
    archive_name_format: '{user}-{hostname}-{now}'

How Long to Keep Backups

# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
# At least one of the "keep" options is required for pruning to work. See
# https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/
# if you'd like to skip pruning entirely.
retention:
    # Keep all archives within this time interval.
    keep_within: 6H

    # Number of hourly archives to keep.
    keep_hourly: 8

    # Number of daily archives to keep.
    keep_daily: 7

    # Number of weekly archives to keep.
    keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

    # Number of yearly archives to keep.
    keep_yearly: 1

    # When pruning, only consider archive names starting with this prefix.
    # Borg placeholders can be used. See the output of "borg help placeholders" for
    # details. Defaults to "{hostname}-". Use an empty value to disable the default.
    prefix: '{user}-{hostname}-'

# Consistency checks to run after backups. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check and
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-extract for details.
consistency:
    # List of one or more consistency checks to run: "repository", "archives", "data",
    # and/or "extract". Defaults to "repository" and "archives". Set to "disabled" to
    # disable all consistency checks. "repository" checks the consistency of the
    # repository, "archives" checks all of the archives, "data" verifies the integrity
    # of the data within the archives, and "extract" does an extraction dry-run of the
    # most recent archive. Note that "data" implies "archives".
    # checks:
        # - repository
        # - archives

    # Paths to a subset of the repositories in the location section on which to run
    # consistency checks. Handy in case some of your repositories are very large, and
    # so running consistency checks on them would take too long. Defaults to running
    # consistency checks on all repositories configured in the location section.
    # check_repositories:
        # - user@backupserver:sourcehostname.borg

    # Restrict the number of checked archives to the last n. Applies only to the "archives" check. Defaults to checking all archives.
    # check_last: 3

    # When performing the "archives" check, only consider archive names starting with
    # this prefix. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Defaults to "{hostname}-". Use an empty
    # value to disable the default.
    prefix: '{user}-{hostname}-'

What To Do on Errors

# Consistency checks to run after backups. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check and
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-extract for details.
consistency:
    # List of one or more consistency checks to run: "repository", "archives", "data",
    # and/or "extract". Defaults to "repository" and "archives". Set to "disabled" to
    # disable all consistency checks. "repository" checks the consistency of the
    # repository, "archives" checks all of the archives, "data" verifies the integrity
    # of the data within the archives, and "extract" does an extraction dry-run of the
    # most recent archive. Note that "data" implies "archives".
    # checks:
        # - repository
        # - archives

    # Paths to a subset of the repositories in the location section on which to run
    # consistency checks. Handy in case some of your repositories are very large, and
    # so running consistency checks on them would take too long. Defaults to running
    # consistency checks on all repositories configured in the location section.
    # check_repositories:
        # - user@backupserver:sourcehostname.borg

    # Restrict the number of checked archives to the last n. Applies only to the "archives" check. Defaults to checking all archives.
    # check_last: 3

    # When performing the "archives" check, only consider archive names starting with
    # this prefix. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Defaults to "{hostname}-". Use an empty
    # value to disable the default.
    prefix: '{user}-{hostname}-'

# Shell commands, scripts, or integrations to execute at various points during a borgmatic
# run. IMPORTANT: All provided commands and scripts are executed with user permissions of
# borgmatic. Do not forget to set secure permissions on this configuration file (chmod
# 0600) as well as on any script called from a hook (chmod 0700) to prevent potential
# shell injection or privilege escalation.
hooks:

    # List of one or more shell commands or scripts to execute when an exception
    # occurs during a "prune", "create", or "check" action or an associated
    # before/after hook.
    # Supported variables:
    #   configuration_filename
    #   repository
    #   error
    #   output
    on_error:
       - ~/.config/borgmatic/notify.sh "{configuration_filename}" "{repository}" "{error}" "{output}"

What To Do Before and After

    # List of one or more shell commands or scripts to execute before running all
    # actions (if one of them is "create"). These are collected from all configuration
    # files and then run once before all of them (prior to all actions).
    # before_everything:
        # - echo "Starting actions."

    # List of one or more shell commands or scripts to execute after running all
    # actions (if one of them is "create"). These are collected from all configuration
    # files and then run once before all of them (prior to all actions).
    # after_everything:
        # - echo "Completed actions."

Secure and Validate Configuration

To prevent potential shell injection or privilege escalation, do not forget to set secure permissions on borgmatic configuration files (chmod 0600) and scripts (chmod 0700) invoked by hooks.

$ chmod 0600 ~/.config/borgmatic/config.yml
$ chmod 0700 ~/.config/borgmatic/notify.sh
$ validate-borgmatic-config

Initialize Repository

$ borgmatic init --encryption keyfile-blake2

Key Files and Passwords

After the initialization a key file is found at ~/.config/borg/keys/$USER.key.

Warning

Without the repository key-file, the repository password and the SSH private keys, your backup data will not be accessible any more.

Store these files and passwords in a save place!

Interactive Backup Test

$ borgmatic --verbosity 1 --files

Systemd Service Files

Service

Start a system backup once, but only if connected to a network and not running on battery.

Systemd service file: ~/.config/systemd/user/borgmatic.service:

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true

[Service]
Type=oneshot

# Lower CPU and I/O priority.
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no
# Prevent rate limiting of borgmatic log events. If you are using an older
# version of systemd that doesn't support this (pre-240 or so), you may have to
# remove this option.
LogRateLimitIntervalSec=0

# Delay start to prevent backups running during boot. Note that systemd-inhibit
# requires dbus and dbus-user-session to be installed.
ExecStartPre=sleep 1m
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /usr/local/bin/borgmatic --syslog-verbosity 1

Schedule

Schedule the system backups once every day at a random time between midnight and 6 AM. or if a scheduled backup time was missed, due to the system powered off or asleep. Don’t wake up the system just for doing a backup.

Systemd timer file ~/.config/systemd/user/borgmatic.timer:

[Unit]
Description=Run borgmatic user data backup

[Timer]
# Schedule the start of the sevice to every day at midnight.
OnCalendar=daily

# Delay the start of the service by a radmon time of up to 6 hours
RandomizedDelaySec=6h

# Only wake up the system for this service after 8 hours or more.
#AccuracySec=8h

# Catch up on missed runs of the service when the system was powered down.
Persistent=true

[Install]
WantedBy=timers.target

Activate

$ systemctl --user enable --now borgmatic.timer