Redis Cluster集群搭建与配置

Redis Cluster是一种服务器sharding分片技术，关于Redis的集群方案应该怎么做，请参考我的另一篇博客http://www.cnblogs.com/xckk/p/6134655.html

本文主要介绍Redis Cluster集群的工作原理，详细讲解了Redis Cluster集群如何搭建与配置。

一、redis安装

redis官网下载后是源码包，需要make安装。

1、解压redis-3.2.5.tar.gz

2、cd redis-3.2.5

3、make

4、cd src/

5、make install

安装完成，src包下面新增redis-server、redis-cli等这些可执行脚本，表示安装完成。

二、启动实例集群配置

搭建Redis Cluster，建立6个redis实例，3主3从，端口号分别为7000-7005

创建以6个端口命名的子目录，之后我们将在6个子目录上都开启Redis实例

cd redis-3.2.

mkdir cluster-test

cd cluster-test

mkdir      

Redis实例配置

在7000到7005每一个目录中都创建redis.conf文件，配置项如下：

cd redis-3.2.

mkdir cluster-test

cd cluster-test

mkdir      

各配置描述如下：

配置项redis.conf	示例	备注
port	7000	端口号
cluster-enabled	yes	yes，表示支持Redis Cluster no,单实例
cluster-config-file	nodes-7000.conf	Redis Cluster记录的启动信息文件，文件由cluster自动生成，不需要用户编辑
cluster-node-timeout	5000	毫秒单位，失效时间
appendonly	yes	支持appendonly持久化方式
protected-mode	no	默认是打开的，此处关闭。因为jedis访问时，如果是保护模式，影响访问
bind	192.168.121.130	机器IP，此处最好写IP，不要写localhost或127.0.0.1，可能影响jedis客户端访问

Redis实例启动

将第一步安装生成的redis-server拷贝到cluster-test目录，分别后台启动6个redis实例

nohup ./redis-server /redis.conf &

nohup ./redis-server /redis.conf &

nohup ./redis-server /redis.conf &

nohup ./redis-server /redis.conf &

nohup ./redis-server /redis.conf &

nohup ./redis-server /redis.conf &

启动完后，cluster-test目录下生成了nodes-7000.conf等启动文件信息。
查看进程启动成功：[root@centos2 cluster-test]# ps -ef|grep redis

三、创建集群

6个redis实例已经创建，它们目前是相互独立的，没有关联。接下来是创建Redis Cluster集群，将6个实例组成一个集群。

redis-trib命令创建集群

src目录下make生成的redis-trib命令，可用于Redis Cluster集群创建的一些脚本启动，它是一个ruby程序。因此我们需要配置ruby环境

安装ruby

yum install ruby

yum install ruby-irb

具体ruby的安装可参考：http://jingyan.baidu.com/article/b7001fe173fe9a0e7382dd57.html

安装Gem

Gem是一个管理Ruby库和程序的标准包，它通过RubyGem（如 http://rubygems.org/ ）源来查找、安装、升级和卸载软件包，非常的便捷。

配置了ruby环境后，还需要安装redis整个包才能够运行redis-trib。安装过程中，需要先安装gem库，不然会出现gem command not found错误。

[root@centos2 src]# gem install redis

-bash: gem: command not found

安装gem过程中，可能出现以下问题

yum install rubygems

ERROR: Could not find a valid gem 'redis' (>= ) in any repository

问题原因：无法连接gem服务器，原因是该gem服务器被墙了。按如下步骤手动下载安装。

wget https://rubygems.global.ssl.fastly.net/gems/redis-3.2.1.gem

gem install -l ./redis-3.2..gem

安装完后，src包下就会生成redis-trib.rb文件

启动命令

./redis-trib.rb create --replicas  192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130:

启动后，各Redis实例之间分别会新增一个端口用于通信，通信端口是Redis实例端口+1W，分别为17000，17001，17002，17003，17004，17005

命令的解释如下：

1、给定的命令将会被ruby程序翻译为create，这表示我们想要创建一个新的集群。

2、选项--replicas 1意思是为每一个master节点创建一个slave节点。

3、其他参数表示redis实例的地址及端口，以空格为间隔。

启动日志分析

出现[OK] All 16384 slots covered.表示启动成功。从启动日志可以看出，master是7000,7001,7002，对应的slave是7003,7004,7005

[root@centos1 src]# ./redis-trib.rb create --replicas  127.0.0.1: 127.0.0.1: 127.0.0.1: 127.0.0.1: 127.0.0.1: 127.0.0.1:

>>> Creating cluster

>>> Performing hash slots allocation on  nodes...

Using  masters:

127.0.0.1:

127.0.0.1:

127.0.0.1:

Adding replica 127.0.0.1: to 127.0.0.1:

Adding replica 127.0.0.1: to 127.0.0.1:

Adding replica 127.0.0.1: to 127.0.0.1:

M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:

slots:- ( slots) master

M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:

slots:- ( slots) master

M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:

slots:- ( slots) master

S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:

replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45

S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:

replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6

S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:

replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251

Can I set the above configuration? (type 'yes' to accept): yes

>>> Nodes configuration updated

>>> Assign a different config epoch to each node

>>> Sending CLUSTER MEET messages to join the cluster

Waiting for the cluster to join...

>>> Performing Cluster Check (using node 127.0.0.1:)

M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:

slots:- ( slots) master

 additional replica(s)

S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:

slots: ( slots) slave

replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251

S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:

slots: ( slots) slave

replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6

M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:

slots:- ( slots) master

 additional replica(s)

S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:

slots: ( slots) slave

replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45

M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:

slots:- ( slots) master

 additional replica(s)

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All  slots covered.

集群启动状态检测

我们可以通过./redis-trib.rb check 192.168.121.130:7000来检测Redis Cluster的启动状态。

[root@centos2 src]# ./redis-trib.rb check 192.168.121.130:

>>> Performing Cluster Check (using node 192.168.121.130:)

M: dca47b274799927fd15aa4e6312b94752418f0c0 192.168.121.130:

slots:- ( slots) master

 additional replica(s)

M: fe1082655e832bd2afb7d4bfabfec3ce2354868b 192.168.121.130:

slots:- ( slots) master

 additional replica(s)

S: 7fe93723c48b7ea33bbf21f12e97d18bdd0221ea 192.168.121.130:

slots: ( slots) slave

replicates dca47b274799927fd15aa4e6312b94752418f0c0

M: 59c926f7bb1e24e293d379507742c52b6dd08cc2 192.168.121.130:

slots:- ( slots) master

 additional replica(s)

S: a554424a9969af2b55f4e11f9eedf1bc3bce9000 192.168.121.130:

slots: ( slots) slave

replicates 59c926f7bb1e24e293d379507742c52b6dd08cc2

S: 065fb213e9db64627c27af93500420351b80ac34 192.168.121.130:

slots: ( slots) slave

replicates fe1082655e832bd2afb7d4bfabfec3ce2354868b

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All  slots covered.

登录集群

通过命令./redis-cli -c -h 192.168.121.129 -p 7000用来登录

重定向

从以下命令可以看出，我们登录的是192.168.121.130机器的7000端口Redis实例，set foo 123，foo经过hash后落到了12182槽，12182槽存储在192.168.121.130机器的7002端口实例，因此日志显示重定向到7002端口的实例。

[root@centos2 src]# ./redis-cli -c -h 192.168.121.130 -p 

192.168.121.130:> set foo 

-> Redirected to slot [] located at 192.168.121.130:

OK

192.168.121.130:> get foo

""

192.168.121.130:>

redis cluster的主从同步采用异步复制（提高性能），因此会存在部分写内容丢失的情况

四、创建集群-通过create-cluster脚本

通过redis-trib.rb命令创建集群涉及命令较多，稍显复杂。好在redis提供了create-cluster脚本utils/create-cluster/create-cluster来创建集群，方便很多。

create-cluster命令介绍

从create-cluster源码看，其实现原理也是利用了redis-trib.rb命令。该脚本默认创建3个master节点，3个salve节点。

1、create-cluster start

创建6个redis实例，默认端口号是30001-30006，如果想改端口号,在create-cluster里面更改PORT值即可。

如果想要端口号是7000-7005，设置PORT=6999即可

2、create-cluster create

创建集群，3个master,3个slave

3、create-cluster stop

关闭Redis Cluster集群和Redis实例

4、create-cluster clean

清除所在Redis实例的data, logs, config文件

使用方法

cd /home/redis-3.2/utils/create-cluster --进入目录

./create-cluster clean ---先清除Redis实例的文件

./create-cluster start ---启动Redis实例

./create-cluster create ---创建Redis Cluster集群

启动完后，使用redis-cli命令登录，即可正常访问。

[root@centos1 create-cluster]# redis-cli -c -p 

127.0.0.1:> set foo hello

-> Redirected to slot [] located at 127.0.0.1:

OK

127.0.0.1:> set hello world

-> Redirected to slot [] located at 127.0.0.1:

OK

127.0.0.1:> get foo

-> Redirected to slot [] located at 127.0.0.1:

"hello"

127.0.0.1:> get hello

-> Redirected to slot [] located at 127.0.0.1:

"world"

127.0.0.1:>

五、Jedis测试程序

Jedis是Redis官方首选的 Java 客户端开发包。我们通过Junit进行单元测试

maven添加jedis和junit依赖包

<dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.8.1</version>
        </dependency>

    </dependencies>

java代码

package com.hk.test.helloWorld;

import java.util.HashSet;
import java.util.List;
import java.util.Set;

import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.junit.Before;
import org.junit.Test;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;

public class TestJedis
{
    JedisCluster jc = null;

    @Before
    public void before()
    {
        Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
        // Jedis Cluster will attempt to discover cluster nodes automatically
        jedisClusterNodes.add(new HostAndPort("centos2", 7000));
        GenericObjectPoolConfig config;
        jc = new JedisCluster(jedisClusterNodes);
    }

    @Test
    public void testFoo()
    {
        for (int i = 0; i <= 3000; i++)
        {
            String key = "foo" + i;
            String value = String.valueOf(i);
            jc.set(key, value);
        }
    }

    @Test
    public void test_incr()
    {

        String key = "page_view";
        jc.del(key);
        jc.incr(key);
        String result = jc.get(key);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetStringVal()
    {
        String key = "foo";
        String value = "bar";
        jc.set(key, value);
        String result = jc.get(key);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetStringVal_and_set_expire() throws InterruptedException
    {
        String key = "hello";
        String value = "world";
        int seconds = 3;
        jc.setex(key, seconds, value);
        String result = jc.get(key);
        System.out.println(result);

        Thread.sleep(seconds * 1000);
        result = jc.get(key);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetHashVal()
    {

        String key = "website";
        String field = "google";
        String value = "google.com";
        jc.del(key);
        jc.hset(key, field, value);
        String result = jc.hget(key, field);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetListVal()
    {

        String key = "mylist";
        jc.del(key);
        String[] vals =
        { "a", "b", "c" };
        jc.rpush(key, vals);
        List<String> result = jc.lrange(key, 0, -1);
        System.out.println(result);

    }

    @Test
    public void test_setAndGetSetVal()
    {

        String key = "language";
        jc.del(key);
        String[] members =
        { "java", "ruby", "python" };
        jc.sadd(key, members);
        Set<String> result = jc.smembers(key);
        System.out.println(result);

    }
}

六、常见问题

一、Could not connect to Redis at 127.0.0.1:30001 Connection refused

原因：create-cluster脚本默认设定的端口从30001开始，由于我设置的redis端口是7000-7005，当时启动cluster时用的redis-trib，因此端口号不同，

./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005

解决方法：

1、create-cluster里面将PORT由30001改为6999。

2、关闭Redis实例进程

可以通过netstat -apn|grep redis或者ps -ef|grep redis查找进程号,kill -9 ｛进程号｝进行关闭。

3、create-cluster stop

4、create-cluster clean 清除data,log,conf文件

5、create-cluster start

6、create-cluster create

二、redis.clients.jedis.exceptions.JedisDataException: DENIED Redis is running in protected mode because protected mode is enabled

原因：Redis实例启动时默认设置了保护模式

解决方法：需要在启动时增加配置项 protected-mode no

1、如果是redis-trib.rb启动，启动配置文件内增加 protected-mode no

port 

cluster-enabled yes

cluster-config-file nodes-.conf

cluster-node-timeout 

appendonly yes

protected-mode no

bind 192.168.121.129

2、如果是create-cluster启动，在create-cluster脚本里start分支中增加protected-mode no

if [ "$1" == "start" ]

then

while [ $((PORT < ENDPORT)) != "" ]; do

PORT=$((PORT+))

echo "Starting $PORT"

../../src/redis-server --port $PORT --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --bind 192.168.121.130 --protected-mode no --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes

done

exit 

fi

三、Jedis访问时，redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?

若用JedisCluster操作集群报以上错误，就是有Redis节点通讯被拒绝，最好是将redis.conf文件中的bind属性设置成redis节点所在服务器的物理ip地址，而不是localhost

1、更改node节点配置文件绑定地址

2、重启redis节点

3、重新create cluster

./redis-trib.rb create --replicas 192.168.121.130: 192.168.121.130: 192.168.121.130: 192.168.121.130:7003192.168.121.130: 192.168.121.130:

四、[ERR] Node 172.168.63.202:7001 is not empty. Either the nodealready knows other nodes (check with CLUSTER NODES) or contains some

解决方法：（一般1，2步即可解决）

1)、将需要新增的节点下aof、rdb等本地备份文件删除；

2)、同时将新Node的集群配置文件删除,即：删除你redis.conf里面cluster-config-file所在的文件；

3)、再次添加新节点如果还是报错，则登录新Node,./redis-cli–h x –p对数据库进行清除：

172.168.63.201:7001>  flushdb      #清空当前数据库

4)、删除后再执行第2,3步，再./create-cluster start即可

五、ERR Invalid node address specified

./redis-trib.rb create --replicas 1 centos2:7000 centos2:7001 centos2:7002 centos2:7003 centos2:7004 centos2:7005

报" ERR Invalid node address specified"

由于Redis-trib.rb 对域名或主机名支持不好,故在创建集群的时候要使用ip:port的方式

./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005

六、ERR Slot 4648 is already busy (Redis::CommandError)

这是由于之前创建集群没有成功,需要

1、将nodes.conf和dir里面的文件全部删除(注意不要删除了redis.conf)

2、将redis实例关闭后重新启动

3、再执行/redis-trib.rb create 。。。脚本

Sorry, the cluster configuration file nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration files.

七、[ERR] Node 127.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解决方法：先stop和clean，再start和create

./create-cluster stop

./create-cluster clean

./create-cluster start

./create-cluster create

秀才坤坤出品

转载请注明来源http://www.cnblogs.com/xckk/p/6144447.html