redis集群之集群搭建

By AverageJoeWang
 标签:

Redis 集群是一个提供在多个Redis间节点间共享数据的程序集.redis3.0以前,只支持主从同步的,如果主的挂了,写入就成问题了。3.0出来后就可以很好帮我们解决这个问题,本文是记录搭建redis集群的过程

一,redis服务器说明

192.168.1.73 6379  
192.168.1.73 6380  
192.168.1.73 6381  

192.168.1.75 6382  
192.168.1.75 6383  
192.168.1.75 6384

要让集群正常运作至少需要三个主节点,不过在刚开始试用集群功能时, 强烈建议使用六个节点: 其中三个为主节点, 而其余三个则是各个主节点的从节点。所有用二台机器,每台机器开3个redis进程,模拟6台机器。

二.安装ruby,rubygems

 yum -y install gcc openssl-devel libyaml-devel libffi-devel readline-devel zlib-devel gdbm-devel ncurses-devel gcc-c++ automake autoconf  

# yum -y install ruby rubygems   //安装ruby rubygems  

//换源  
# gem source -l  
# gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/
# gem source -l  

# gem install redis --version 3.2.2
Fetching: redis-3.2.2.gem (100%)
Successfully installed redis-3.2.2
Parsing documentation for redis-3.2.2
Installing ri documentation for redis-3.2.2
1 gem installed

三.安装与配置redis-3.2.2

3.1.安装redis

# wget http://download.redis.io/releases/redis-3.2.2.tar.gz  
# tar -xvzf redis-3.2.2.tar.gz  
# cd redis-3.2.2  
# make && make install  
# cd src  
# cp redis-trib.rb /usr/local/bin    
# mkdir /etc/redis  
# mkdir /var/log/redis

上述操作要先在二台机器上面都操作好。

3.2.配置redis

[root@localhost redis-3.2.2]#  vim redis.conf   //redis解压的根目录,有redis.conf,做以下修改 
daemonize yes
port 6379  
pidfile /var/run/redis-6379.pid  
dbfilename dump-6379.rdb  
appendfilename "appendonly-6379.aof"  
cluster-config-file nodes-6379.conf  
cluster-enabled yes  
cluster-node-timeout 5000  
appendonly yes

3.3.配置6个配置文件

# cp redis_6379.conf redis_XXXX.conf
# vim redis_XXXX.conf
#XXXX指的是4位的端口号
port XXXX  
pidfile /var/run/redis-XXXX.pid  
dbfilename dump-XXXX.rdb  
appendfilename "appendonly-XXXX.aof"  
cluster-config-file nodes-XXXX.conf
  • 利用vim的全局替换命令
:%s/6379/6381/g

# sed -i "s/6379/6380/g" /etc/redis/redis_6380.conf  
# sed -i "s/6379/6381/g" /etc/redis/redis_6381.conf  

192.168.1.75服务器

# sed -i "s/6379/6382/g" /etc/redis/redis_6382.conf  
# sed -i "s/6379/6383/g" /etc/redis/redis_6383.conf  
# sed -i "s/6379/6384/g" /etc/redis/redis_6384.conf

3.4.copy配置文件到相应目录

# cp redis_6379.conf /etc/redis/redis_6379.conf  
# cp redis_6380.conf /etc/redis/redis_6380.conf  
# cp redis.conf_6381 /etc/redis/redis_6381.conf  

# scp redis_6382.conf 192.168.1.75:/etc/redis/redis_6382.conf  
# scp redis_6383.conf 192.168.1.75:/etc/redis/redis_6383.conf  
# scp redis_6384.conf 192.168.1.75:/etc/redis/redis_6384.conf

3.5.替换端口

[root@localhost redis]# cat redis_6384.conf |awk '{if($0 !~ /^$/ && $0 !~ /#/) {print $0}}' |grep 6384
port 6384
pidfile /var/run/redis_6384.pid
dbfilename dump_6384.rdb
appendfilename "appendonly_6384.aof"
cluster-config-file nodes-6384.conf
#一共5处需要修改端口

3.6.启动并且查看redis

  • 启动
# redis-server /etc/redis/redis_6379.conf > /var/log/redis/redis_6379.log 2>&1 &  
# redis-server /etc/redis/redis_6380.conf > /var/log/redis/redis_6380.log 2>&1 &  
# redis-server /etc/redis/redis_6381.conf > /var/log/redis/redis_6381.log 2>&1 &  
# netstat -tpnl |grep redis
# ll /etc/redis/

# redis-server /etc/redis/redis_6382.conf > /var/log/redis/redis_6382.log 2>&1 &  
# redis-server /etc/redis/redis_6383.conf > /var/log/redis/redis_6383.log 2>&1 &  
# redis-server /etc/redis/redis_6384.conf > /var/log/redis/redis_6384.log 2>&1 &
# netstat -tpnl |grep redis
# ll /etc/redis/
  • 反馈

192.168.1.73服务器

[root@localhost redis]# redis-server /etc/redis/redis_6379.conf > /var/log/redis/redis_6379.log 2>&1 &  
[1] 8581
[root@localhost redis]# redis-server /etc/redis/redis_6380.conf > /var/log/redis/redis_6380.log 2>&1 &  
[2] 8584
[root@localhost redis]# redis-server /etc/redis/redis_6381.conf > /var/log/redis/redis_6381.log 2>&1 &  
[3] 8587
[root@localhost redis]# netstat -tpnl |grep redis
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      8581/redis-server 1 
tcp        0      0 127.0.0.1:6380          0.0.0.0:*               LISTEN      8584/redis-server 1 
tcp        0      0 127.0.0.1:6381          0.0.0.0:*               LISTEN      8587/redis-server 1 
tcp        0      0 127.0.0.1:16379         0.0.0.0:*               LISTEN      8581/redis-server 1 
tcp        0      0 127.0.0.1:16380         0.0.0.0:*               LISTEN      8584/redis-server 1 
tcp        0      0 127.0.0.1:16381         0.0.0.0:*               LISTEN      8587/redis-server 1 
[root@localhost redis]# ll /etc/redis
total 156
-rw-r--r--. 1 root root     0 Dec 14 11:45 appendonly_6379.aof
-rw-r--r--. 1 root root     0 Dec 14 11:46 appendonly_6380.aof
-rw-r--r--. 1 root root     0 Dec 14 11:46 appendonly_6381.aof
-rw-r--r--. 1 root root   112 Dec 14 11:45 nodes-6379.conf
-rw-r--r--. 1 root root   112 Dec 14 11:46 nodes-6380.conf
-rw-r--r--. 1 root root   112 Dec 14 11:46 nodes-6381.conf
-rw-r--r--. 1 root root 46700 Dec 14 11:38 redis_6379.conf
-rw-r--r--. 1 root root 46700 Dec 14 11:38 redis_6380.conf
-rw-r--r--. 1 root root 46700 Dec 14 11:38 redis_6381.conf

192.168.1.75服务器

[root@localhost redis]# redis-server /etc/redis/redis_6382.conf > /var/log/redis/redis_6382.log 2>&1 &
[1] 11931
[root@localhost redis]# redis-server /etc/redis/redis_6383.conf > /var/log/redis/redis_6383.log 2>&1 &
[2] 11934
[root@localhost redis]# redis-server /etc/redis/redis_6384.conf > /var/log/redis/redis_6384.log 2>&1 &
[3] 11937
[root@localhost redis]# netstat -tpnl |grep redis  
tcp        0      0 127.0.0.1:6382          0.0.0.0:*               LISTEN      11931/redis-server  
tcp        0      0 127.0.0.1:6383          0.0.0.0:*               LISTEN      11934/redis-server  
tcp        0      0 127.0.0.1:6384          0.0.0.0:*               LISTEN      11937/redis-server  
tcp        0      0 127.0.0.1:16382         0.0.0.0:*               LISTEN      11931/redis-server  
tcp        0      0 127.0.0.1:16383         0.0.0.0:*               LISTEN      11934/redis-server  
tcp        0      0 127.0.0.1:16384         0.0.0.0:*               LISTEN      11937/redis-server  
[root@localhost redis]# ll /etc/redis/
total 156
-rw-r--r--. 1 root root     0 Dec 13 22:46 appendonly_6382.aof
-rw-r--r--. 1 root root     0 Dec 13 22:46 appendonly_6383.aof
-rw-r--r--. 1 root root     0 Dec 13 22:47 appendonly_6384.aof
-rw-r--r--. 1 root root   112 Dec 13 22:46 nodes-6382.conf
-rw-r--r--. 1 root root   112 Dec 13 22:46 nodes-6383.conf
-rw-r--r--. 1 root root   112 Dec 13 22:47 nodes-6384.conf
-rw-r--r--. 1 root root 46700 Dec 13 22:39 redis_6382.conf
-rw-r--r--. 1 root root 46700 Dec 13 22:39 redis_6383.conf
-rw-r--r--. 1 root root 46700 Dec 13 22:39 redis_6384.conf

所有节点都启动成功,但是目前还不是集群

四.创建集群并且查看

4.1.创建集群

redis-trib.rb create --replicas 1 192.168.1.73:6379 192.168.1.73:6380 192.168.1.73:6381 192.168.1.75:6382 192.168.1.75:6383 192.168.1.75:6384
  • 结果
[root@localhost redis]# redis-trib.rb create --replicas 1 192.168.1.73:6379 192.168.1.73:6380 192.168.1.73:6381 192.168.1.75:6382 192.168.1.75:6383 192.168.1.75:6384
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.73:6379
192.168.1.75:6382
192.168.1.73:6380
Adding replica 192.168.1.75:6383 to 192.168.1.73:6379
Adding replica 192.168.1.73:6381 to 192.168.1.75:6382
Adding replica 192.168.1.75:6384 to 192.168.1.73:6380
M: a17a886f4ce0509767fb33efb5f55f5a5fbf17fe 192.168.1.73:6379
   slots:0-5460 (5461 slots) master
M: 9d2d90222bd14903e970816591e5f8badeffeacd 192.168.1.73:6380
   slots:10923-16383 (5461 slots) master
S: d4d177a13ae4adc9d24c8862eb423c51493237b1 192.168.1.73:6381
   replicates 7d3ea2ae244afc76f2e6edc58a901a8015ada0a0
M: 7d3ea2ae244afc76f2e6edc58a901a8015ada0a0 192.168.1.75:6382
   slots:5461-10922 (5462 slots) master
S: 35d4dd22abe659e38cd64f58620afe26da10c749 192.168.1.75:6383
   replicates a17a886f4ce0509767fb33efb5f55f5a5fbf17fe
S: 5b964c670ddc53e12869bdb2dd39764f0d911c53 192.168.1.75:6384
   replicates 9d2d90222bd14903e970816591e5f8badeffeacd
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....
>>> Performing Cluster Check (using node 192.168.1.73:6379)
M: a17a886f4ce0509767fb33efb5f55f5a5fbf17fe 192.168.1.73:6379
   slots:0-5460 (5461 slots) master
M: 9d2d90222bd14903e970816591e5f8badeffeacd 192.168.1.73:6380
   slots:10923-16383 (5461 slots) master
M: d4d177a13ae4adc9d24c8862eb423c51493237b1 192.168.1.73:6381
   slots: (0 slots) master
   replicates 7d3ea2ae244afc76f2e6edc58a901a8015ada0a0
M: 7d3ea2ae244afc76f2e6edc58a901a8015ada0a0 192.168.1.75:6382
   slots:5461-10922 (5462 slots) master
M: 35d4dd22abe659e38cd64f58620afe26da10c749 192.168.1.75:6383
   slots: (0 slots) master
   replicates a17a886f4ce0509767fb33efb5f55f5a5fbf17fe
M: 5b964c670ddc53e12869bdb2dd39764f0d911c53 192.168.1.75:6384
   slots: (0 slots) master
   replicates 9d2d90222bd14903e970816591e5f8badeffeacd
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

五.问题与解决方案

5.1.问题一

问题重现

[err]: Test replication partial resync: ok psync (diskless: yes, reconnect: 1) in tests/integration/replication-psync.tcl
Expected condition '[s -1 sync_partial_ok] > 0' to be true ([s -1 sync_partial_ok] > 0)
Cleanup: may take some time... OK
make[1]: *** [test] Error 1
make[1]: Leaving directory `/usr/local/src/redis-3.2.1/src'
make: *** [test] Error 2

原因

是因为在redis等待超时引起的,网上一共有2种解决方案

解决方案

  • 只用单核运行make test
taskset -c 1 sudo make test
  • 更改tests/integration/replication-psync.tcl 文件
vi tests/integration/replication-psync.tcl

把对应报错的那段代码中的 after后面的数字,从100改成 500。这个参数就是等待的毫秒数。

修改之后再make test就会发现原来之前错误是由于等待超时的结果。

Execution time of different units:
  0 seconds - unit/printver
  0 seconds - unit/type/incr
  0 seconds - unit/scan
  1 seconds - unit/auth
  1 seconds - unit/quit
  1 seconds - unit/protocol
  1 seconds - unit/keyspace
  2 seconds - unit/type/hash
  5 seconds - unit/type/zset
  5 seconds - unit/type/set
  6 seconds - unit/sort
  6 seconds - unit/multi
  1 seconds - integration/convert-zipmap-hash-on-load
  1 seconds - integration/rdb
  0 seconds - integration/logging
  1 seconds - unit/pubsub
  8 seconds - unit/other
  8 seconds - unit/type/string
  0 seconds - unit/introspection
  1 seconds - unit/slowlog
  4 seconds - integration/aof
  2 seconds - unit/limits
  10 seconds - unit/type/list-2
  10 seconds - unit/expire
  1 seconds - unit/bitfield
  4 seconds - unit/maxmemory
  12 seconds - unit/type/list
  3 seconds - unit/bitops
  12 seconds - integration/replication-2
  3 seconds - unit/memefficiency
  6 seconds - unit/scripting
  7 seconds - unit/introspection-2
  6 seconds - unit/geo
  9 seconds - unit/hyperloglog
  21 seconds - integration/replication-3
  23 seconds - unit/dump
  16 seconds - unit/obuf-limits
  28 seconds - integration/replication-4
  42 seconds - unit/aofrw
  92 seconds - unit/type/list-3
  116 seconds - integration/replication
  274 seconds - integration/replication-psync

5.2.问题二

问题重现can't connect to node

redis-trib.rb create --replicas 1 192.168.1.73:6379 192.168.1.73:6380 192.168.1.73:6381 192.168.1.75:6382 192.168.1.75:6383 192.168.1.75:6384后出现

[ERR] Sorry, can't connect to node 192.168.1.73:6379

原因与解决方案

是因为redis服务在配置文件/etc/redis_XXXX.conf中是默认bind 127.0.0.1的,这个需要在127.0.0.1前面加上前面的那个本机ip就能够解决这个问题

六.参考

1.RubyGems 镜像
2.问题解决方案
3.Redis集群教程
4.Redis集群搭建
5.vim替换命令