Redis

Redis Logo

Redis is a fast and persistent key-value database with a network interface.

As a key-value database it is similar to server/memcache but the dataset is not only stored entirely in memory but periodically flushed to disk. This way the dataset remains persistent across restarts and is no longer only volatile.

It supports many kind of data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps and more.

It has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

Topology

The following server applications in our environment, use Redis to store and retrieve data fast in memory:

  • Rspamd for …
    • Ratelimit plugin uses Redis to store limits buckets;
    • Greylisting module stores data and meta hashes inside Redis;
    • DMARC module can save DMARC reports inside Redis keys;
    • Replies plugin requires Redis to save message ids hashes for outgoing messages;
    • IP score plugin uses Redis to store data about AS, countries and networks reputation;
    • Multimap module can use Redis as readonly database for maps;
    • MX Check module uses Redis for caching;
  • Rspamd statistics module to store Bayes tokens;
  • Rspamd storage for fuzzy hashes;
  • nextcloud-server for transactional file locking;

For Rspamd we set up a master/slave replication. This way all mail servers can share their knowledge about connecting SMTP servers and spaminess of messages.

The mail server will act as master, while the the remote MX hosts will act as slaves:

         +------------------+
         |    Charlotte     |
         |  (Redis Master)  |
         |    Nextcloud     |
         |    Postfix       |
         |    Dovecot       |
         |    Rspamd        |
         +------------------+
            |             |
            |             |
+-----------------+  +-----------------+
|     Dolores     |  |      Maeve      |
|  (Redis Slave)  |  |  (Redis Slave)  |
|     Postfix     |  |     Postfix     |
|     Rspamd      |  |     Rspamd      |
+-----------------+  +-----------------+
Host Role IPv4 VPN Address IPv6 VPN Address
Charlotte Master 10.195.171.241 fdc1:d89e:b128:6a04::29ab
Dolores Slave 10.195.171.142 fdc1:d89e:b128:6a04::7de4
Maeve Slave 10.195.171.47 fdc1:d89e:b128:6a04::961

It is strongly recommended not to share a Redis database among different applications and although Redis has limited support for multiple databases, it is also strongly recommended by its author not to use this feature. Instead one should run a dedicated instance of Redis with a single dedicated database for every application.

Considering our list of applications above, we need four Redis instances, of which 3 are gonna to be replicated across three servers. A total of 10 instances to setup across all servers.

Application Instance TCP Port Mode of Operation
Nextcloud Server nextcloud 6380 Standalone
Rspamd rspamd 6381 Master/Slave
Rspamd Statistics Module rspamd-bayes 6382 Master/Slave
Rspamd Fuzzy Storage Worker rspamd-fuzzy 6383 Master/Slave

Prerequisites

WireGuard VPN

The data stored in Redis by Rspamd will be replicated across remote connected servers. Since Redis does not provide any security by itself, we use Virtual Private Network connections between the masters and slaves.

System Control

Background saves might fail under low memory conditions. It therefore might help to tell Linux to allocate memory more aggressively:

$ sudo sysctl vm.overcommit_memory=1

To make this persistent across reboots:

$ echo "vm.overcommit_memory=1" | sudo tee /etc/sysctl.d/60-overcommit-mem.conf
$ sudo systemctl restart procps.service

Kernel Configuration

The Linux Kernel feature “Transparent Huge Pages (THP)” has negative effects on memory usage and latency of Redis. Therefore it should be disabled:

$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

Installation

Redis is installed from packages:

$ sudo apt install redis-server

Configuration

Redis configuration is stored in /etc/redis.conf. Most of the settings can be left at their default. We use a include statement to add the small number of settings which sets each instance apart.

Common Configuration

In the file /etc/redis/common.conf we set all the default settings, which are common to all of our Redis instances. This file can then be included on each instance configuration file, which keeps the whole thing more manageable and less cluttered.

We use the default file as template:

$ sudo mv /etc/redis/redis.conf /etc/redis/common.conf

Note

Make sure the file-name does not contain the string redis, because if it does, systemd will try to start an instance with this file as configuration.

Things to do here:

  • Comment out any “bind” statements, as they are defined in the instance configuration files;
  • Comment out any “unixsocket” statements.
  • Turn off protected mode, as we will use networking and password protection later;
#bind 127.0.0.1 ::1
# unixsocket /var/run/redis/redis-server.sock
# unixsocketperm 700
protected-mode no

Nextcloud Standalone Instance

Things to do here:

  • Common configuration include statement;
  • Set memory limit;
  • Set localhost address and TCP port;
  • Set a password for the local client(s);

Create a password for Nextcloud:

$ pwgen --secure 32 1
Wk7H302Kx4JxW0Beb2WIcvHbLbpRnkse

/etc/redis/redis-nextcloud.conf:

#
# Redis configuration file on Charlotte
# for Nextcloud server transactional file locking.
#

include /etc/redis/redis.conf
# Localhost
bind 127.0.0.1 ::1
port 6380
unixsocket /var/run/redis-nextcloud/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-nextcloud/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-nextcloud.log
dbfilename dump-nextcloud.rdb
requirepass Wk7H302Kx4JxW0Beb2WIcvHbLbpRnkse
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Rspamd Master Instance

Things to do here:

  • Common configuration include statement;
  • Set memory limit;
  • Set Wireguard VPN interface address and TCP port;
  • Set a password for slaves and local clients;

Create a password for Rspamd:

$ pwgen --secure 32 1
CyJGIROS8CA15qITGd1vZKsPQb6ESwYV

/etc/redis/redis-rspamd.conf:

#
# Redis Master configuration on Charlotte
# for Rspamd Modules
# 	Ratelimit, Greylisting, DMARC, Replies, IP score, Multimap, MX Check
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.241 fdc1:d89e:b128:6a04::29ab
port 6381
unixsocket /var/run/redis-rspamd/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-rspamd/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd.log
dbfilename dump-rspamd.rdb
requirepass CyJGIROS8CA15qITGd1vZKsPQb6ESwYV
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Rspamd Slave Instance

Things to do here:

  • Common configuration include statement;
  • Set memory limit;
  • Set Wireguard VPN interface address and TCP port;
  • Set password for local clients;
  • Set address of the master;
  • Set password to access the master;

/etc/redis/redis-rspamd.conf:

#
# Redis Slave configuration on Dolores
# for Rspamd Modules
# 	Ratelimit, Greylisting, DMARC, Replies, IP score, Multimap, MX Check
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.142 fdc1:d89e:b128:6a04::7de4
port 6381
unixsocket /var/run/redis-rspamd/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /run/redis-rspamd/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd.log
dbfilename dump-rspamd.rdb
requirepass CyJGIROS8CA15qITGd1vZKsPQb6ESwYV
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5
slaveof charlotte.vpn.example.net 6381
masterauth CyJGIROS8CA15qITGd1vZKsPQb6ESwYV

Rspamd Bayesian Master Instance

Things to do here:

  • Common configuration include statement;
  • Set memory limit;
  • Set Wireguard VPN interface address and TCP port;
  • Set a password for slaves and local clients;

Create a password for Rspamd:

$ pwgen --secure 32 1
WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p

/etc/redis/redis-rspamd-bayes.conf:

#
# Redis Master configuration on Charlotte
# for Rspamd Bayesian statistic Module
#

include /etc/redis/common.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.241 fdc1:d89e:b128:6a04::29ab
port 6382
unixsocket /var/run/redis-rspamd-bayes/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-rspamd-bayes/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd-bayes.log
dbfilename dump-rspamd-bayes.rdb
requirepass WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5

Rspamd Bayesian Slave Instance

Things to do here:

  • Common configuration include statement;
  • Set memory limit;
  • Set Wireguard VPN interface address and TCP port;
  • Set password for local clients;
  • Set address of the master;
  • Set password to access the master;

/etc/redis/redis-rspamd-bayes.conf:

#
# Redis Slave configuration on Dolores
# for Rspamd Bayesian statistic Module
#

include /etc/redis/redis.conf
# Listen on localhost and WireGuard VPN Interface
bind 127.0.0.1 ::1 10.195.171.142 fdc1:d89e:b128:6a04::7de4
port 6382
unixsocket /var/run/redis-rspamd-bayes/redis-server.sock
unixsocketperm 700
daemonize yes
supervised systemd
pidfile /var/run/redis-rspamd-bayes/redis-server.pid
loglevel notice
logfile /var/log/redis/redis-rspamd-bayes.log
dbfilename dump-rspamd-bayes.rdb
requirepass WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p
# maxmemory <bytes>
# maxmemory-policy noeviction
# maxmemory-samples 5
slaveof charlotte.vpn.example.net 6382
masterauth WRTgpIv6Xz6i7lNNMQ13bijAj5ghPR7p

Systemd Services

Systemd has a built in support to automatically start multiple instances of a service. This needs only a couple of small changes in the service file.

Make a copy of the pre-installed service file /lib/systemd/system/redis-server@.service:

$ sudo cp /lib/systemd/system/redis-server@.service /etc/systemd/system/

Note

On some systems I discovered an additional service file /etc/systemd/system/redis.service. Please remove it, if thats the case.

Edit the file /etc/systemd/system/redis-server.service, as follows:

# Templated service file for redis-server(1)
#
# Each instance of redis-server requires its own configuration file:
#
#   $ cp /etc/redis/redis.conf /etc/redis/redis-myname.conf
#   $ chown redis:redis /etc/redis/redis-myname.conf
#
# Ensure each instance is using their own database:
#
#   $ sed -i -e 's@^dbfilename .*@dbfilename dump-myname.rdb@' /etc/redis/redis-myname.conf
#
# We then listen exlusively on UNIX sockets to avoid TCP port collisions:
#
#   $ sed -i -e 's@^port .*@port 0@' /etc/redis/redis-myname.conf
#   $ sed -i -e 's@^\(# \)\{0,1\}unixsocket .*@unixsocket /var/run/redis-myname/redis-server.sock@' /etc/redis/redis-myname.conf
#
# ... and ensure we are logging, etc. in a unique location:
#
#   $ sed -i -e 's@^logfile .*@logfile /var/log/redis/redis-server-myname.log@' /etc/redis/redis-myname.conf
#   $ sed -i -e 's@^pidfile .*@pidfile /var/run/redis-myname/redis-server.pid@' /etc/redis/redis-myname.conf
#
# We can then start the service as follows, validating we are using our own
# configuration:
#
#   $ systemctl start redis-server@myname.service
#   $ redis-cli -s /var/run/redis-myname/redis-server.sock info | grep config_file
#
#  -- Chris Lamb <lamby@debian.org>  Mon, 09 Oct 2017 22:17:24 +0100
[Unit]
Description=Advanced key-value store (%I)
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)

[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis-%i.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/var/run/redis-%i/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis-%i
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/redis-%i

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# redis-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target

Since we don’t use any clustering features of Redis, we can restrict the systemd services a bit, by keeping the configuration read-only:

# redis-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
#ProtectSystem=true
#ReadWriteDirectories=-/etc/redis
ProtectSystem=full

If you do this, you have to make sure the configuration files always remain readable by the redis user. I.e. after you edited the configuration files as root:

$ sudo chown -Rc redis:redis /etc/redis
$ sudo systemctl daemon-reload
$ sudo systemctl restart redis-server.service