Redis

Redis is a fast and persistent key-value database with a network interface.

As a key-value database it is similar to Memcache but the dataset is not only stored entirely in memory but periodically flushed to disk. This way the dataset remains persistent across restarts and is no longer only volatile.

It supports many kind of data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps and more.

It has built-in replication, LUA scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

Topology 

The following server applications in our environment, use Redis to store and retrieve data fast in memory:

Rspamd Spam Filter for … * Ratelimit plugin uses Redis to store limits buckets; * Greylisting module stores data and meta hashes inside Redis; * DMARC module can save DMARC reports inside Redis keys; * Replies plugin requires Redis to save message ids hashes for outgoing messages; * IP score plugin uses Redis to store data about AS, countries and networks reputation; * Multimap module can use Redis as readonly database for maps; * MX Check module uses Redis for caching;

Rspamd Spam Filter statistics module to store Bayes tokens;

Rspamd Spam Filter storage for fuzzy hashes;

Postfix TLS policies of domains who publish a MTA-STS policy;

Nextcloud for transactional file locking;

For Rspamd we set up a master/slave replication. This way all mail servers can share their knowledge about connecting SMTP servers and spaminess of messages.

The mail server will act as master, while the the remote MX hosts will act as slaves:

         +------------------+
         |    Charlotte     |
         |  (Redis Master)  |
         |    Nextcloud     |
         |    Postfix       |
         |    Dovecot       |
         |    Rspamd        |
         +------------------+
            |             |
            |             |
+-----------------+  +-----------------+
|     Dolores     |  |      Maeve      |
|  (Redis Slave)  |  |  (Redis Slave)  |
|     Postfix     |  |     Postfix     |
|     Rspamd      |  |     Rspamd      |
+-----------------+  +-----------------+

Host	Role	IPv4 VPN Address	IPv6 VPN Address
Charlotte	Master	10.195.171.241	fdc1:d89e:b128:6a04::29ab
Dolores	Slave	10.195.171.142	fdc1:d89e:b128:6a04::7de4
Maeve	Slave	10.195.171.47	fdc1:d89e:b128:6a04::961

It is strongly recommended not to share a Redis database among different applications and although Redis has limited support for multiple databases, it is also strongly recommended by its author not to use this feature. Instead one should run a dedicated instance of Redis with a single dedicated database for every application.

Considering our list of applications above, we need five Redis instances, of which four are gonna to be replicated across three servers. A total of 13 instances to setup across all servers.

Application	Instance	TCP Port	Mode of Operation
Nextcloud Server	nextcloud	6380	Standalone
Rspamd	rspamd	6381	Master/Slave
Rspamd Statistics Module	rspamd-bayes	6382	Master/Slave
Rspamd Fuzzy Storage Worker	rspamd-fuzzy	6383	Master/Slave
Postfix MTA-STS TLS policy	postfix-tls	6384	Master/Slave

Prerequisites 

WireGuard VPN 

The data stored in Redis by Rspamd will be replicated across remote connected servers. Since Redis does not provide any security by itself, we use Virtual Private Network connections between the masters and slaves.

System Control 

Background saves might fail under low memory conditions. It therefore might help to tell Linux to allocate memory more aggressively:

$ sudo sysctl vm.overcommit_memory=1

To make this persistent across reboots:

$ echo "vm.overcommit_memory=1" | sudo tee /etc/sysctl.d/60-overcommit-mem.conf
$ sudo systemctl restart procps.service

Kernel Configuration 

The Linux Kernel feature “Transparent Huge Pages (THP)” has negative effects on memory usage and latency of Redis. Therefore it should be disabled:

$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

Installation 

Redis is installed from packages:

$ sudo apt install redis-server

Configuration 

Redis configuration is stored in /etc/redis.conf. Most of the settings can be left at their default. We use a include statement to add the small number of settings which sets each instance apart.

Common Configuration 

In the file /etc/redis/common.conf we set all the default settings, which are common to all of our Redis instances. This file can then be included on each instance configuration file, which keeps the whole thing more manageable and less cluttered.

We use the default file as template:

$ sudo mv /etc/redis/redis.conf /etc/redis/common.conf

Note

Make sure the file-name does not contain the string redis, because if it does, systemd will try to start an instance with this file as configuration.

Things to do here:

Comment out any “bind” statements, as they are defined in the instance configuration files;

Comment out any “unixsocket” statements.

Turn off protected mode, as we will use networking and password protection later;

#bind 127.0.0.1 ::1

# unixsocket /var/run/redis/redis-server.sock
# unixsocketperm 700

protected-mode no

Nextcloud Standalone Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set localhost address and TCP port;

Set a password for the local client(s);

Create a password for Nextcloud:

$ pwgen --secure 32 1
Wk7H302Kx4JxW0Beb2WIcvHbLbpRnkse

/etc/redis/redis-nextcloud.conf:

#
# Redis configuration file on Charlotte
# for Nextcloud server transactional file locking.
#

include /etc/redis/redis.conf
# Localhost
bind 127.0.0.1 ::1
port 6380
unixsocket /var/run/redis-nextcloud/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-nextcloud/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-nextcloud.log
dbfilename dump-nextcloud.rdb
requirepass Wk7H302Kx4JxW0Beb2WIcvHbLbpRnkse
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Rspamd Master Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set Wireguard VPN interface address and TCP port;

Set a password for slaves and local clients;

Create a password for Rspamd:

$ pwgen --secure 32 1
CyJGIROS8CA15qITGd1vZKsPQb6ESwYV

/etc/redis/redis-rspamd.conf:

#
# Redis Master configuration on Charlotte
# for Rspamd Modules
# 	Ratelimit, Greylisting, DMARC, Replies, IP score, Multimap, MX Check
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.241 fdc1:d89e:b128:6a04::29ab
port 6381
unixsocket /var/run/redis-rspamd/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-rspamd/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd.log
dbfilename dump-rspamd.rdb
requirepass CyJGIROS8CA15qITGd1vZKsPQb6ESwYV
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Rspamd Slave Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set Wireguard VPN interface address and TCP port;

Set password for local clients;

Set address of the master;

Set password to access the master;

/etc/redis/redis-rspamd.conf:

#
# Redis Slave configuration on Dolores
# for Rspamd Modules
# 	Ratelimit, Greylisting, DMARC, Replies, IP score, Multimap, MX Check
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.142 fdc1:d89e:b128:6a04::7de4
port 6381
unixsocket /var/run/redis-rspamd/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /run/redis-rspamd/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd.log
dbfilename dump-rspamd.rdb
requirepass CyJGIROS8CA15qITGd1vZKsPQb6ESwYV
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5
slaveof charlotte.vpn.example.net 6381
masterauth CyJGIROS8CA15qITGd1vZKsPQb6ESwYV

Rspamd Bayesian Master Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set Wireguard VPN interface address and TCP port;

Set a password for slaves and local clients;

Create a password for Rspamd:

$ pwgen --secure 32 1
WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p

/etc/redis/redis-rspamd-bayes.conf:

#
# Redis Master configuration on Charlotte
# for Rspamd Bayesian statistic Module
#

include /etc/redis/common.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.241 fdc1:d89e:b128:6a04::29ab
port 6382
unixsocket /var/run/redis-rspamd-bayes/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-rspamd-bayes/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd-bayes.log
dbfilename dump-rspamd-bayes.rdb
requirepass WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Rspamd Bayesian Slave Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set Wireguard VPN interface address and TCP port;

Set password for local clients;

Set address of the master;

Set password to access the master;

/etc/redis/redis-rspamd-bayes.conf:

#
# Redis Slave configuration on Dolores
# for Rspamd Bayesian statistic Module
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.142 fdc1:d89e:b128:6a04::7de4
port 6382
unixsocket /var/run/redis-rspamd-bayes/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-rspamd-bayes/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd-bayes.log
dbfilename dump-rspamd-bayes.rdb
requirepass WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5
slaveof charlotte.vpn.example.net 6382
masterauth WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p

Postfix MTA-STS TLS Master Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set Wireguard VPN interface address and TCP port;

Set password for local clients;

Set address of the master;

Set password to access the master;

Create a password for Postifx:

$ pwgen --secure 32 1
ZlsQPlZAwMRpBgzEvwH2J7jsWkcpC7Xr

/etc/redis/redis-postfix-tls.conf:

#
# Redis Master configuration on Charlotte
# for Postifx MTA-STS TLS Policy
#

include /etc/redis/common.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.241 fdc1:d89e:b128:6a04::29ab
port 6384
unixsocket /var/run/redis-postfix-tls/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-postfix-tls/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-postfix-tls.log
dbfilename dump-postfix-tls.rdb
requirepass ZlsQPlZAwMRpBgzEvwH2J7jsWkcpC7Xr
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Postfix MTA-STS TLS Slave Instance 

Things to do here:

Common configuration include statement;

Set memory limit;

Set Wireguard VPN interface address and TCP port;

Set password for local clients;

Set address of the master;

Set password to access the master;

/etc/redis/redis-postfix-tls.conf:

#
# Redis Slave configuration on Dolores
# for Postifx MTA-STS TLS Policy
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.142 fdc1:d89e:b128:6a04::7de4
port 6384
#unixsocket /run/redis-postfix-tls/redis-server.sock
#unixsocketperm 700
daemonize yes
supervised systemd
pidfile /run/redis-postfix-tls/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-postfix-tls.log
dbfilename dump-postfix-tls.rdb
requirepass ZlsQPlZAwMRpBgzEvwH2J7jsWkcpC7Xr
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5
slaveof charlotte.vpn.example.net 6384
masterauth ZlsQPlZAwMRpBgzEvwH2J7jsWkcpC7Xr

Systemd Services 

Systemd has a built in support to automatically start multiple instances of a service. The service file /lib/systemd/system/redis-server@.service installed by the software package already contains everything to start a separate service instance for each redis-*.conf file found in the /etc/redis/ directory.

However, since some of our Redis instances are set to replicate their data over the WireGuard VPN to other servers, we need to make sure that the VPN interface is up and running, before these Redis server instances are started.

The following Redis instances replicate data over the VPN:

postfix-tls

rspamd

rspamd-bayes

rspamd-fuzzy

You can create a Systemd override file easily with the help of the systemctl command:

$ sudo systemctl edit redis-server@postfix-tls.service

This will start your editor with an empty file, where you can add your own custom Systemd service configuration options.

[Unit]
After=sys-devices-virtual-net-wg0.device
BindsTo=sys-devices-virtual-net-wg0.device

[Service]
ProtectSystem=full

The line After=sys-devices-virtual-net-wg0.device ensures that wg0 network interface is up before the postfix-tls Redis server instance is started.

The line BindsTo=sys-devices-virtual-net-wg0.device ensures that if the wg0 network interface goes down, this Redis server instance will be stopped too.

Since we don’t use any clustering features of Redis, we can restrict the systemd services a bit, by keeping the configuration read-only. this is what the line ProtectSystem=full does.

After you save and exit the editor (CTRL+X, Y), the file will be saved as /etc/systemd/system/mariadb.service.d/override.conf and Systemd will reload its configuration.

Repeat the procedure for the Rspamd Redis server instances:

$ sudo systemctl edit redis-server@rspamd.service
$ sudo systemctl edit redis-server@rspamd-bayes.service
$ sudo systemctl edit redis-server@rspamd-fuzzy.service

Make a copy of the pre-installed service file /lib/systemd/system/redis-server@.service:

$ sudo cp /lib/systemd/system/redis-server@.service /etc/systemd/system/

Note

On some systems I discovered an additional service file /etc/systemd/system/redis.service. Please remove it, if that’s the case.

Edit the file /etc/systemd/system/redis-server@.service, as follows:

# Templated service file for redis-server(1)
#
# Each instance of redis-server requires its own configuration file:
#
#   $ cp /etc/redis/redis.conf /etc/redis/redis-myname.conf
#   $ chown redis:redis /etc/redis/redis-myname.conf
#
# Ensure each instance is using their own database:
#
#   $ sed -i -e 's@^dbfilename .*@dbfilename dump-myname.rdb@' /etc/redis/redis-myname.conf
#
# We then listen exlusively on UNIX sockets to avoid TCP port collisions:
#
#   $ sed -i -e 's@^port .*@port 0@' /etc/redis/redis-myname.conf
#   $ sed -i -e 's@^\(# \)\{0,1\}unixsocket .*@unixsocket /var/run/redis-myname/redis-server.sock@' /etc/redis/redis-myname.conf
#
# ... and ensure we are logging, etc. in a unique location:
#
#   $ sed -i -e 's@^logfile .*@logfile /var/log/redis/redis-server-myname.log@' /etc/redis/redis-myname.conf
#   $ sed -i -e 's@^pidfile .*@pidfile /var/run/redis-myname/redis-server.pid@' /etc/redis/redis-myname.conf
#
# We can then start the service as follows, validating we are using our own
# configuration:
#
#   $ systemctl start redis-server@myname.service
#   $ redis-cli -s /var/run/redis-myname/redis-server.sock info | grep config_file
#
#  -- Chris Lamb <lamby@debian.org>  Mon, 09 Oct 2017 22:17:24 +0100
[Unit]
Description=Advanced key-value store (%I)
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)
Wants=sys-devices-virtual-net-wg0.device
After=sys-devices-virtual-net-wg0.device

[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis-%i.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/var/run/redis-%i/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis-%i
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/redis-%i

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# redis-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target

Since we don’t use any clustering features of Redis, we can restrict the systemd services a bit, by keeping the configuration read-only:

# redis-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
#ProtectSystem=true
#ReadWriteDirectories=-/etc/redis
ProtectSystem=full

If you do this, you have to make sure the configuration files always remain readable by the redis user. I.e. after you edited the configuration files as root:

$ sudo chown -Rc redis:redis /etc/redis

References 

Rspamd Replication in Redis

Redis Configuration

Running Multiple Redis Instances