RabbitMQ脑裂

在RabbitMQ3.4.x中会出现脑裂的现象，本文通过实验验证此脑裂现象，愿小伙伴们少走弯路。

Preview

网上有两篇帖子（需要FQ）

https://groups.google.com/forum/#!topic/rabbitmq-users/dt8VFhMb2zM

https://groups.google.com/forum/#!topic/rabbitmq-users/06OQkYtLJd8

陈述了脑裂的现象。

帖子中描述现象：

Hey Folk,

i just set up a rabbitmq cluster:

Three Nodes:
Node A | Node B | Node C

All three nodes see each other (same erlang-cookie, mode: pause_minority).
 rabbitmqctl cluster_status => shows status of all nodes on every instance.

Every queue is mirrored to the other nodes.

If i shutdown Node B, the following is happening:
* Node A realizes Node B is offline.
* Node A asks Node C for Node B status.
* Node C answers: "I still have connection to Node B."
* Node A shuts down itself.
* Node C realizes some seconds later, that the connection to Node B is no more possible.

From three Nodes only one is left in case of an unexpected outage.

I would like to realize a setup where Node A and C keep the connection even if Node B goes offline.
Is there any way to do this?

Michael Klishin（rabbitmq-server第二贡献者）回复：

A known issue which is partially resolve in 3.4.x releases. 26474 can be related.

(根据RabbitMQ 3.4.2 Release日志：26474 prevent false positive detection of partial partitions (since 3.4.0))

Simon MacMullen(也是rabbitmq-server的contributor)：

So this is caused by the new partial partition detection in 3.4.x. It
looks like it is too sensitive - C should only reply "yes" if it has
positive confirmation that it can still talk to B, not if the connection
just hasn't failed yet. 

This will be fixed in 3.4.2.

假设

自此可以假设：rabbitmq3.4.0存在脑裂现象，rabbitmq3.4.2修复了此bug。

论证过程：分别对rabbitmq3.4.0, rabbitmq3.4.1, rabbitmq3.4.2, rabbitmq3.6.0进行实验, 分别配置A B C三个节点组成一个cluster，然后通过停止C的网络来验证A和B是否出现脑裂.

可以通过发现脑裂问题证明rabbitmq3.4.0和rabbitmq3.4.1有bug，而无法通过没发现脑裂问题证明rabbitmq3.4.2之后的版本没有此bug.

就好像证明一个男人是坏男人，一次出轨就可以说明了；要证明一个男人是好男人，要盯着他一辈子看他有没有出轨……

论证

论证1

rabbitmq版本:3.4.0

rabbitmq节点配置

共三个节点：A B C，分别为：

A:rabbit@zhuzhonghua2-fqawb

B:rabbit@hiddenzhu-8drd

C:rabbit@hidden-local

B join_cluster A; C join_cluster A

查看cluster_status:(rabbitmqctl cluster_status)

Cluster status of node 'rabbit@zhuzhonghua2-fqawb' ...
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                 'rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

在C节点执行service network stop

在A节点查看cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

再次在A节点查看cluster_status

Cluster status of node 'rabbit@zhuzhonghua2-fqawb' ...
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hiddenzhu-8drdc']}]}]
在B节点查看cluster_status
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

结论：【复现脑裂】

在C节点执行service network start

查看A节点cluster_status

[{nodes,
     [{disc,
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
           'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,
     [{'rabbit@zhuzhonghua2-fqawb',
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

查看C节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hidden-local',['rabbit@zhuzhonghua2-fqawb']}]}]

论证2

rabbitmq版本：3.4.1

节点配置如上（B join_cluster A, C join_cluster A）

查看节点状态：

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb',
                 'rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

在C节点执行service network stop

查看A节点cluster_status

Cluster status of node 'rabbit@zhuzhonghua2-fqawb' ...
[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hiddenzhu-8drdc']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

结论：【复现脑裂】

在C节点执行service network start

查看A节点cluster_status

[{nodes,
     [{disc,
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
           'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,
     [{'rabbit@zhuzhonghua2-fqawb',
          ['rabbit@hidden-local','rabbit@hiddenzhu-8drdc']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hiddenzhu-8drdc',['rabbit@zhuzhonghua2-fqawb']}]}]

查看C节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hidden-local',['rabbit@zhuzhonghua2-fqawb']}]}]

论证3

rabbitmq版本：3.4.2 （版本3.6.0与此相同）

节点配置如上（B join_cluster A, C join_cluster A）

查看节点状态

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb',
                 'rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

在C节点执行service network stop

查看A节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb','rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[]}]

结论：【未发现脑裂】

在C节点执行service network start

查看A节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hiddenzhu-8drdc','rabbit@zhuzhonghua2-fqawb']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hidden-local']}]}]

查看B节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@zhuzhonghua2-fqawb','rabbit@hiddenzhu-8drdc']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@zhuzhonghua2-fqawb',['rabbit@hidden-local']}]}]

查看C节点cluster_status

[{nodes,[{disc,['rabbit@hidden-local','rabbit@hiddenzhu-8drdc',
                'rabbit@zhuzhonghua2-fqawb']}]},
 {running_nodes,['rabbit@hidden-local']},
 {cluster_name,<<"rabbit@zhuzhonghua2-fqawb">>},
 {partitions,[{'rabbit@hidden-local',['rabbit@zhuzhonghua2-fqawb']}]}]

结论

版本问题基本得到验证，为了防止脑裂现象，建议正在使用rabbitmq的小伙伴升级，避免使用3.4.0和3.4.1这两个版本。

但是依然会有网络分区的问题！！！！

网络分区

有关网络分区有篇文章（RabbitMQ 网络分区问题）这样介绍：

RabbitMQ 集群的网络分区容错性并不是非常高，在网络经常发生分区时会有些问题，最明显的就是脑裂问题。

官方文档是这样介绍的：

RabbitMQ clusters do not tolerate network partitions well. If you are thinking of clustering across a WAN, don't. You should use federation or the shovel instead.

从中我们可以看出，在广域网环境下不应该使用集群，而应该使用 federation 或者 shovel 来解决。

不过即使是在局域网环境下，网络分区也不可能完全避免，网络设备（比如中继设备、网卡）出现故障也会导致网络分区。

Network partition detected

Mnesia reports that this RabbitMQ cluster has experienced a network partition. This is a dangerous situation. RabbitMQ clusters should not be installed on networks which can experience partitions.

当出现网络分区时，不同分区里的节点会认为不属于自身所在分区的节点都已经挂了，对 queue、exchange、binding 的操作仅对当前分区有效。在 RabbitMQ 的默认配置下，即使网络恢复了也不会自动处理网络分区带来的问题从而恢复集群。RabbitMQ（3.1+）会自动探测网络分区，并且提供了配置来解决这个问题。

[
 {rabbit,
 [{tcp_listeners,[5672]},
 {cluster_partition_handling, ignore}]
 }
].

RabbitMQ 提供了三种配置：

ignore：默认配置，发生网络分区时不作处理，当认为网络是可靠时选用该配置
autoheal：各分区协商后重启客户端连接最少的分区节点，恢复集群（CAP 中保证 AP，有状态丢失）
pause_minority：分区发生后判断自己所在分区内节点是否超过集群总节点数一半，如果没有超过则暂停这些节点（保证 CP，总节点数为奇数个）

参考：

● RabbitMQ 官方文档

● 网络分区

● 脑裂问题

RabbitMQ脑裂的更多相关文章

RabbitMq脑裂问题
现象部署在阿里云上的2台RabbitMQ主从,访问management页面时出现如下所示的内容: 查看其中一个mq的日志,发现如下内容: 00:06:32.423 [warning] <0.5 ...
RabbitMQ脑裂问题解决方案调查
现象: RabbitMQ GUI上显示 Network partition detectedMnesia reports that this RabbitMQ cluster has experien ...
rabbitmq 脑裂(网络分区)
1.产生的原因 https://blog.csdn.net/zyz511919766/article/details/45198055 2.相关配置.如何规避 https://blog.csdn.ne ...
[译]如何防止elasticsearch的脑裂问题
本文翻译自blog.trifork.com的博文地址是http://blog.trifork.com/2013/10/24/how-to-avoid-the-split-brain-problem- ...
如何防止ElasticSearch集群出现脑裂现象（转）
原文:http://xingxiudong.com/2015/01/05/resolve-elasticsearch-split-brain/ 什么是“脑裂”现象? 由于某些节点的失效,部分节点的网络 ...
高可用性中的脑裂问题(split-brain problem in HA)(转)
欢迎关注我的社交账号: 邮箱: jiangxinnju@163.com 博客园地址: http://www.cnblogs.com/jiangxinnju GitHub地址: https://gith ...
Zookeeper 脑裂
转自 http://blog.csdn.net/u010185262/article/details/49910301 Zookeeper zookeeper是一个分布式应用程序的协调服务.它是一个为 ...
AIX下解决POWERHA的脑裂问题
一.安装创建并发vg时必需的软件包clvm包,该包安装.升级.后必须重启os clvm包的描述:Enhanced Concurrent Logical Volume Manager 软件包在aix61 ...
Elasticsearch笔记八之脑裂
Elasticsearch笔记八之脑裂概述: 一个正常es集群中只有一个主节点,主节点负责管理整个集群,集群的所有节点都会选择同一个节点作为主节点所以无论访问那个节点都可以查看集群的状态信息. 而脑 ...

随机推荐

uva656 Optimal Programs
Optimal Programs As you know, writing programs is often far from being easy. Things become even hard ...
[转]通过apk签名使应用程序有系统权限
[转]通过apk签名使应用程序有系统权限 (2013-01-08 13:40:50) 转载▼ 标签: it 分类: Android 出处:http://blog.csdn.net/doom66151/ ...
ObjectId
BSON Types — MongoDB Manual https://docs.mongodb.com/manual/reference/bson-types/#objectid ObjectId ...
cloud native
什么是原生云(cloud native)应用? 原生云cloud-native应用的定义是:首先,应用系统应该与底层物理基础设施解耦.说白了,应用程序应该与操作系统等基础设施分离,不应该依赖Linux ...
d3.js 之增加感染力：使用转场效果
转场/transition 图形比数据有感染力,动起来的图形比静态的图形更有感染力. 转场是一种过渡,提供两个稳定状态间的一种动态渐进的变化.转场的概念来源于电影. 电影中存在不同场景之间的切换,比如 ...
Secure Sockets Layer(安全套接层)
SSL SSL(Secure Sockets Layer安全套接层)及其继任者传输层安全(Transport Layer Security,TLS)是为网络通信提供安全及数据完整性的一种安全协议.TL ...
Python-读入json文件并进行解析及json基本操作
import json def resolveJson(path): file = open(path, "rb") fileJson = json.load(file) fi ...
zabbix详解（一）
zabbix简介 zabbix是一个基于WEB界面的提供分布式系统监视以及网络监视功能的企业级的开源解决方案. zabbix能监视各种网络参数,保证服务器系统的安全运营:并提供柔软的通知机制以让系统管 ...
PAT 1039 Course List for Student[难]
1039 Course List for Student (25 分) Zhejiang University has 40000 students and provides 2500 courses ...
MySQL整理（三）
一.简单单表操作 (1)简单CRUD 插入查询结果 insert into table1(id,name,age) select id,name,age from table2 where id ...

RabbitMQ脑裂

Preview

假设

论证

论证1

论证2

论证3

结论

网络分区

RabbitMQ脑裂的更多相关文章

随机推荐

热门专题