Redis Sentinel 模拟故障迁移

什么是redis sentinel

参考文档:https://redis.io/topics/sentinel

简单的来说，就是Redis Sentinel 为redis 提供高可用性，主要体现在下面几个方面:
1.监控:redis sentinel会不间断的监控主服务器和从服务器是否正常工作
2.通知:当出现问题时，sentinel可以通过API通知系统管理员以及另外的服务器
3.自动故障转移:如果主服务器出现故障，sentinel可以启动故障转移，将其中一台从服务器升级为主服务器，其他的从服务器会重新配置为新主服务器 4.提供配置:sentinel充当客户端发现权限来源，客户端连接到sentinel询问负责给定服务器当前redis主服务器地址，如果发生故障，sentinel将报告新地址

redis sentinel 模拟环境

模拟环境为:1主2从

========redis=================sentinel==========

master:127.0.0.1 6379       127.0.0.1 26379
slave1:127.0.0.1 6380       127.0.0.1 26380
slave2:127.0.0.1 6381       127.0.0.1 26381

环境搭建

redis.conf配置

6379

# .conf | grep -Ev "^$|^#"
bind 127.0.0.1
port
daemonize yes
pidfile /var/run/redis_6379.pid
logfile "/root/redis/redis-6379.log"
dbfilename dump-.rdb
dir /root/redis
...
#

6380

# .conf | grep -Ev "^$|^#"
bind 127.0.0.1
port
daemonize yes
pidfile /var/run/redis_6380.pid
logfile "/root/redis/redis-6380.log"
dbfilename dump-.rdb
dir /root/redis
...
#

6381

# .conf | grep -Ev "^$|^#"
bind 127.0.0.1
port
daemonize yes
pidfile /var/run/redis_6381.pid
logfile "/root/redis/redis-6381.log"
dbfilename dump-.rdb
dir /root/redis
...
#

sentinel.conf配置

6379/6380/6381

# cat sentinel-*.conf | grep -Ev "^#|^$"
port
daemonize yes
logfile "/root/redis/sentinel-6379.log"
dir "/tmp"
sentinel monitor mymaster
sentinel down-after-milliseconds mymaster
sentinel parallel-syncs mymaster
sentinel failover-timeout mymaster
#

启动redis server 和 sentinel

redis:
# redis-server /etc/redis_6379.conf
# redis-server /etc/redis_6380.conf
# redis-server /etc/redis_6381.conf

sentinel:
# redis-sentinel /etc/sentinel-.conf
# redis-sentinel /etc/sentinel-.conf
# redis-sentinel /etc/sentinel-.conf

配置主从复制

# redis-cli -p
> SLAVEOF
OK
> exit

# redis-cli -p
> SLAVEOF
OK
> exit

模拟故障迁移

首先，kill 掉redis master进程

#  |  $n  ;done;

分析log

首先，redis 从服务器首先发现redis master 服务器无法连接，报错如下:

# *.log
==> redis-.log <==
:S  Nov ::54.235 # Connection with master lost.
:S  Nov ::54.235 * Caching the disconnected master state.

==> redis-.log <==
:S  Nov ::
:S  Nov ::54.466 * MASTER <-> SLAVE sync started
:S  Nov ::54.467 # Error condition on socket for SYNC: Connection refused

==> redis-.log <==
:S  Nov ::
:S  Nov ::54.782 * MASTER <-> SLAVE sync started
:S  Nov ::54.782 # Error condition on socket for SYNC: Connection refused
...

紧接着,redis sentinel 完成故障切换,从log来看，当6379主节点挂了之后，redis重新提了一个从节点6380为主节点，log 如下:

# *.log
==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov :: #quorum /
:X  Nov ::
:X  Nov ::

==> sentinel-.log <==
:X  Nov :: #quorum /
:X  Nov ::
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::

==> sentinel-.log <==
:X  Nov ::
:X  Nov ::
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster
:X  Nov ::
:X  Nov ::  @ mymaster 

==> sentinel-.log <==
:X  Nov :: @ mymaster
:X  Nov ::
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster 

==> sentinel-.log <==
:X  Nov :: @ mymaster
:X  Nov ::
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster 

==> sentinel-.log <==
:X  Nov ::
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster
:X  Nov ::
:X  Nov ::
:X  Nov ::  @ mymaster
:X  Nov ::  @ mymaster 

==> sentinel-.log <==
:X  Nov ::  @ mymaster 

==> sentinel-.log <==
:X  Nov ::  @ mymaster 

==> sentinel-.log <==
:X  Nov ::  @ mymaster 

再返回过来看redis server的log,此时可以看到6381为从节点已经向主节点6380请求并且完成了复制操作

==> redis-.log <==
:M  Nov :: asks for synchronization
:M  Nov :: accepted. Sending  bytes of backlog starting from offset .

==> redis-.log <==
:S  Nov ::25.823 * Successful partial resynchronization with master.
:S  Nov ::25.823 # Master replication ID changed to 0288d040464ebccbb56dc56d54455434a406bcb2
:S  Nov ::25.823 * MASTER <-> SLAVE sync: Master accepted a Partial Resynchronization.

当我们再启动6379服务器时，sentinel会让6379成为从库并且连接6380服务器,log如下: 

启动6379服务器
# redis-server /root/redis/redis-.conf

# *.log
...
==> sentinel-.log <==
:X  Nov ::  @ mymaster
...

# *.log
...
==> redis-.log <==
:S  Nov :: bytes from master
:S  Nov ::00.566 * MASTER <-> SLAVE sync: Flushing old data
:S  Nov ::00.566 * MASTER <-> SLAVE sync: Loading DB in memory
:S  Nov ::00.566 * MASTER <-> SLAVE sync: Finished with success

==> redis-.log <==
:S  Nov :: changes  seconds. Saving...
:S  Nov ::
:C  Nov ::36.486 * DB saved on disk
:C  Nov :: MB of memory used by copy-on-write
:S  Nov ::36.569 * Background saving terminated with success
...

未完待续。。。