Mon blog-notes à moi que j'ai

Blog personnel d'un sysadmin, tendance hacker

Discovering Redis Cluster 3.0

The 6th Release Candidate of Redis 3 is out. Amongst all feature, one is particularily awaited: cluster mode. How does it works and what can be done with it ?

Introduction

Feel free to read the official documentation

I’ll give here some explanation, mostly from the official documentation. I hope you’ll be able to better understand what’s going on with Redis Cluster.

Quorum

To avoid a network partition, you should use a odd number of servers. This allow the cluster to always get a majority when making decisions.

Data replication

It’s not mandatory, but you should also setup one slave per master so that you can tolerate losing one master without loosing data. Some software like Elasticsearch use shard replication between master. In Redis Cluster, there’s no cross master replication

Data storage

Previous Redis versions allowed a multi DB feature. This feature is not available anymore in Redis Cluster.
Redis Cluster uses only one DB. Data are spreaded into 16384 hash slots. Destination slot is calculated as follow

HASH_SLOT = CRC16(key) mod 16384

hash slots are spreaded across master nodes.

By default, Redis uses key full name to determine destination hash slot. You can shortcut that behaviour using { and } inside key name.
As an example, following keys will be stored into the same slot

{key::name}::value
foo::{key::name}::value

This is because brackets tell Redis which part of the key name must be used to compute destination hash slot.

Redis commands scope is bounded by slot. That means you can’t perform cross-slot operation. When manipulating bitmaps, you most likely want to be able to execute BITOP command. For that to work, you’ll have to ensure that all the keys are stored in the same slot and that’s where comes key hash tags.

Cluster topology

Each master node keep a copy of the whole cluster topology.
Cluster’s nodes will get synchronized through the network cluster bus. It uses, by default, TCP/16379.

This connection uses Redis listening port plus 10000. This value is hardcoded.
Therefore, you can not start a Redis cluster node listening on a port greater than 55536

Every node of a n nodes cluster maintains n-1 incoming, and outgoing, connections, to maintain a unified cluster topology. Redis cluster is a full mesh cluster.

Usage

Create a cluster

The minimal configuration file for a Redis cluster is as follow

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-require-full-coverage yes
cluster-migration-barrier 1
cluster-slave-validity-factor 10
appendonly yes

To start Redis, just execute

redis-server redis7000.conf

Redis new listens on port TCP/7000 for Redis itself, and TCP/17000 for cluster bus.
Redis cluster won’t allocate any slots. You’ll have to do it manually (do not do it right now, juste read bellow :) )

redis-cli -p 7000 CLUSTER ADDSLOTS 0
redis-cli -p 7000 CLUSTER ADDSLOTS 1
redis-cli -p 7000 CLUSTER ADDSLOTS 2
redis-cli -p 7000 CLUSTER ADDSLOTS 3
[...]
redis-cli -p 7000 CLUSTER ADDSLOTS 16383

OK, let’s avoid doing all that manually. Redis provides a Ruby script which allows you to easily bootstrap a Redis cluster (in my case, I installed the script in /usr/bin).

/usr/bin/redis-trib.rb fix 127.0.0.1:7000

This tools will chek cluster slots allocation status and will offer to you to fix what has to be fixed.

Of course, a one node cluster is not very usefull. Let’s add another node

port 7002
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-require-full-coverage yes
cluster-migration-barrier 1
cluster-slave-validity-factor 10
appendonly yes

Start the second instance and make it join the cluster

redis-server redis7002.conf
redis-cli -p 7002 CLUSTER MEET 127.0.0.1 7000

Let’s check cluster status

/usr/bin/redis-trib check 127.0.0.1:7000
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7002: OK
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 1327103b8e5b4921bf47941943c4b279a50d8ed5 127.0.0.1:7000
   slots:0-16383 (16384 slots) master
   0 additional replica(s)
M: 6a5a36527d49b1c21275feffa0f4c3b6b62b49c5 127.0.0.1:7002
   slots: (0 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

New node has been added, but does not manage any slots. You’ll need to fix it manually. In our case, we’ll move half of the slots on the second node.

/usr/bin/redis-trib reshard 127.0.0.1:7000
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7002: OK
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 1327103b8e5b4921bf47941943c4b279a50d8ed5 127.0.0.1:7000
   slots:0-16383 (16384 slots) master
   0 additional replica(s)
M: 6a5a36527d49b1c21275feffa0f4c3b6b62b49c5 127.0.0.1:7002
   slots: (0 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 8192
What is the receiving node ID? 6a5a36527d49b1c21275feffa0f4c3b6b62b49c5
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:1327103b8e5b4921bf47941943c4b279a50d8ed5
Source node #2:done
[...]

Once done, we can check the clusetr again

/usr/bin/redis-trib check 127.0.0.1:7000
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7002: OK
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 1327103b8e5b4921bf47941943c4b279a50d8ed5 127.0.0.1:7000
   slots:8192-16383 (8192 slots) master
   0 additional replica(s)
M: 6a5a36527d49b1c21275feffa0f4c3b6b62b49c5 127.0.0.1:7002
   slots:0-8191 (8192 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Now, add a third node on port TCP/7004. I’ll let you do the maths needed to spread the slots :)

/usr/bin/redis-trib.rb check 127.0.0.1:7000
Connecting to node 127.0.0.1:7000: OK
Connecting to node 127.0.0.1:7004: OK
Connecting to node 127.0.0.1:7002: OK
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 9b6e118f13081f6c8ded4d2419cdb21a0ad1bbc7 127.0.0.1:7000
   slots:8192-16383 (8192 slots) master
   0 additional replica(s)
M: 978ec5bc35cf57332d0c0672a7ff3f2fe3a43666 127.0.0.1:7004
   slots: (0 slots) master
   0 additional replica(s)
M: 1dfc1b008c56c8b9c415d930b7e53f4677f8b1bd 127.0.0.1:7002
   slots:0-8191 (8192 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Once you have nodes id, you can spread slots in a non interactive way

/usr/bin/redis-trib.rb reshard 127.0.0.1:7000 \
                       --from 9b6e118f13081f6c8ded4d2419cdb21a0ad1bbc7 \
                       --to 978ec5bc35cf57332d0c0672a7ff3f2fe3a43666 \
                       --slots 2730 --yes
/usr/bin/redis-trib.rb reshard 127.0.0.1:7000 \
                       --from 1dfc1b008c56c8b9c415d930b7e53f4677f8b1bd \
                       --to 978ec5bc35cf57332d0c0672a7ff3f2fe3a43666 \
                       --slots 2731 --yes

Once done, you can add a slave to each master.
Create configuration files for slave using ports TCP/7001, TCP/7003 and TCP/7005 and start them. Then, you can make them join the cluster as slave

/usr/bin/redis-trib add-node --slave 127.0.0.1:7001 127.0.0.1:7000
/usr/bin/redis-trib add-node --slave 127.0.0.1:7003 127.0.0.1:7002
/usr/bin/redis-trib add-node --slave 127.0.0.1:7005 127.0.0.1:7004

Now you have a fully functional cluster. I’ll let you test failover. you just have to kill a master process.

Enjoy :)