Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention certain kinds of failures.
Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.
This is the full list of Sentinel capabilities at a macroscopical level (i.e. the big picture):
- Monitoring. Sentinel constantly checks if your master and replica instances are working as expected.
- Notification. Sentinel can notify the system administrator, or other computer programs, via an API, that something is wrong with one of the monitored Redis instances.
- Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a replica is promoted to master, the other additional replicas are reconfigured to use the new master, and the applications using the Redis server are informed about the new address to use when connecting.
- Configuration provider. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.
How sentinel works
Sentinels handle the failover by re-writing config files of the Redis instances that are running. Let’s go through a scenario:
Say we have a master “A” replicating to slaves “B” and “C”. We have three Sentinels (s1, s2, s3) running on our application servers, which write to Redis. At this point “A”, our current master, goes offline. Our sentinels all see “A” as offline, and send SDOWN messages to each other. Then they all agree that “A” is down, so “A” is set to be in ODOWN status. From here, an election happens to see who is most ahead, and in this case “B” is chosen as the new master.
The config file for “B” is set so that it is no longer the slave of anyone. Meanwhile, the config file for “C” is rewritten so that it is no longer the slave of “A” but rather “B.” From here, everything continues on as normal. Should “A” come back online, the Sentinels will recognize this, and rewrite the configuration file for “A” to be the slave of “B,” since “B” is the current master.
Before installation, redis must be install into the system. In this example, we’ll create one master node of redis and one slave which will be managed by one sentinel called as s1.
Redis master configuration
Just create a master.conf file which is a duplicate file of redis.conf which is in redis installation directory. After create master.conf just change the following things:
# Run as background service daemonize yes # for daemonized pid file path pidfile "/var/run/redis_6379.pid" # specify log file path logfile "/var/log/6379.log" # Set the number of databases, from 0 to 16 databases 16 # specify db dump path dbfilename dump.rdb # db will written into this directory dir "/var/dump/" # specify redis instance port port 6379 # bind your public/local ip, to let you access by your ip address bind ip1 ip2 ip
After the settings of master.conf file just start the redis server with this configuration file as below:
Start Master node
redis-cli -h 127.0.0.1 -p 6379 info replication
After executing this command you will see following output:
# Replication role:master connected_slaves:0 master_replid:049098811534d348dd88d82df0672295ee261482 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:179479 master_repl_meaningful_offset:179479 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:179479
As you can see that there is no connected_slaves as of now with this master redis. But this has been configured as master in the environment.
Now let’s configure slave node of redis as below:
For configuring slave node, you need to change the properites like bind ip, port, log directory, dump file etc as you did in master.conf. Apart from that below configuration needs to change in order to make redis node as slave of master node.
# Specify master instance location replicaOf 127.0.0.1 6379
Now its time to start slave as below:
Start Slave Node
Suppose you given slave port as 6380 so it will start on this port.
Verify master and slave node
Let’s verify whether it has been configured with master as slave or not by executing following command again:
redis-cli -h 127.0.0.1 -p 6379 info replication
It will show us following output:
# Replication role:master connected_slaves:1 slave0:ip=127.0.0.1,port=6380,state=online,offset=199429,lag=1 master_replid:049098811534d348dd88d82df0672295ee261482 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:199429 master_repl_meaningful_offset:199429 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:199429
As you can see, connected_slaves is 1 which means that slave node has been configured successfully.
It’s time to configure sentinel to manage both the nodes. Just open sentinel.conf file and change following things:
# port to run sentinel port 26379 dir /tmp # Consider master objectively down only if at least number of sentinels agree sentinel monitor master 127.0.0.1 6379 1 # How many seconds master not responding will consider it as down sentinel down-after-milliseconds master 3000 # How many slave could sentinel reconfigure simultaneously, use low number to prevent all instance unreachable at about the same time sentinel parallel-syncs master 1 # Sentinel failover timeout sentinel failover-timeout master 18000
redis-server sentinel.conf --sentinel
redis-cli -h 127.0.0.1 -p 26379 info sentinel
It will show you following output:
# Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=1,sentinels=1
As you can see the information about slaves and masters.
It’s time to verify the sentinel and replication of things:
Just Execute the following command:
redis-cli -h 127.0.0.1 -p 6379 127.0.0.1:6379> hset first-key hello world integer 1 127.0.0.1:6379>keys * 1) "first-key" ### After that just connect with the slave node in order to check the replication of data into slave node redis-cli -h 127.0.0.1 -p 6380 127.0.0.1:6380> keys * 1) "first-key"
As you can see that we create a set into master node and it is automatically gets created or replicated into slave node.
To know more about sentinel click here.