keepalived+双主实践HA
正好前段时间看过关于keepalived+双主实现高可用的文章,也恰好身边的朋友所在的公司也部分用这个架构。对比下MMM、MHA、keepalived+双主三种架构的优劣和DB维护的体验感。简单讲讲自己的用户体验感,就搭建难易程度讲MMM的安装包封装好的,修改的配置文件较MHA少一些,比keepalived+双主要稍微麻烦点儿。本着省事,维护起来省事还是觉得MMM真的便利,黑盒操作适合我这种懒人加小白类型。
某位大佬讲过,如果你的公司还在用MMM和MHA,那么你可以考虑跳槽了。我觉得这句话很有道理,其实并不是让我们真的去跳槽,毕竟每个人工作的目的,环境不一样,有些架构上的事儿我们决定不了。没法随性而行,但不能停止探索的脚步,本过程从搭建调研/搭建过程/搭建测试/搭建总结四个方面讲述我对双主+keepalived的理解和用户体验感。
一、搭建调研
传统的高可用架构如MHA、MMM存在一些不成熟的问题,如脑裂。引入keepalived和双主复制模式,实现高可用架构,但keepalived本身是在机器宕机时才会实现漂移功能,我们的目标是要MySQL实例宕机后要实现故障切换,还需要辅助的脚本来帮助keepalived来实现更灵活的漂移。
keepalived简介
keepalived是集群管理中保证集群高可用的一个软件解决方案,其功能类似于heartbeat,用来防止单点故障,这里的作用我理解其实就是保证VIP的顺利漂移。虚拟路由冗余协议,可以认为是实现路由器高可用的协议,即将N台提供相同功能的路由器组成一个路由器组,这个组里面有一个master和多个backup,master上面有一个对外提供服务的vip,master会发组播(组播地址为224.0.0.18),当backup收不到vrrp包时就认为master宕掉了,这时就需要根据VRRP的优先级来选举一个backup当master,这样的话就可以保证路由器的高可用了。
keepalived配置说明
keepalived只有一个配置文件keepalived.conf,里面主要包括以下几个配置区域,分别是global_defs、vrrp_instance和virtual_server。
- global_defs:主要是配置故障发生时的通知对象以及机器标识;
- vrrp_instance:用来定义对外提供服务的VIP区域及其相关属性;
- virtual_server:虚拟服务器定义。
二、搭建过程
搭建环境(服务器配置忽略)
master1:172.16.3.190/22 3309 VIP:172.16.3.123/22
#master1配置keepalived
yum install keepalived.x86_64
[root@--- we_ops_admin]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs {
router_id lvs_master1
} vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id
priority
advert_int
nopreempt
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
172.16.3.123/
}
} virtual_server 172.16.3.123 {
delay_loop
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout
protocol TCP real_server 172.16.3.190 {
weight
notify_down /opt/shells/keepalived_mysql.sh
TCP_CHECK {
connect_timeout
nb_get_retry
delay_before_retry
connect_port
}
}
} #master2上安装keepalived
yum install keepalived.x86_64
[root@--- we_ops_admin]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs {
router_id lvs_master2
} vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id
priority
advert_int
# nopreempt
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
172.16.3.123/
}
} virtual_server 172.16.3.123 {
delay_loop
lb_algo rr
lb_kind NAT
nat_mask 255.255.255.0
persistence_timeout
protocol TCP real_server 172.16.3.189 {
weight
notify_down /opt/shells/keepalived_mysql.sh
TCP_CHECK {
connect_timeout
nb_get_retry
delay_before_retry
connect_port
}
}
}
上述配置中我们可以保证keepalived服务对VIP:172.16.3.123/22的控制权,默认是keepalived服务关闭,那么会触发VIP的漂移。正常运行的服务不会发生异常停止的现象,如果系统发生宕机会触发所有的服务停止,这里系统宕机是触发VIP漂移的导火索。只是这里我们想让keepalived服务于MySQL复制集,那么这里的导火索自然而然是MySQL服务的状态。如果服务状态不可用,那么我们希望这个应用VIP可以漂移到复制集的另一台机器上;如果服务状态可用,我们希望VIP不要漂移。要想实现这个目的,我们还需要一个服务脚本来帮助我们去帮助keepalived发现MySQL服务宕机后的动作,脚本如下配置。
[root@--- we_ops_admin]# cat /opt/shells/keepalived_mysql.sh
#!/bin/bash
pkill keepalived
/sbin/ifdown eth0 && /sbin/ifup eth0
#授予可执行权限
[root@--- we_ops_admin]# ls -lh /opt/shells/keepalived_mysql.sh
-rwxr-xr-x root root Sep : /opt/shells/keepalived_mysql.sh
通过步骤1·2的配置,启动MySQL服务,启动keepalived服务,这里的master1和master2基本就可以实现高可用,保证了master1服务不可用时,master2还能继续提供数据库的支持。
三、搭建测试(Bash脚本模拟高并发)
1、master1的MySQL服务宕机,VIP会从master1上摘除漂移落盘到master2上,且master1上的keepalived服务也会停止。应用连接VIP,master2继续为整个集群提供数据库支持。
#停止master1上的MySQL服务
[root@--- we_ops_admin]# /etc/init.d/mysql_3309 stop
Shutting down MySQL (Percona Server).. SUCCESS! #keepalived服务也停止了,且VIP已经被从master1上摘除
[root@--- we_ops_admin]# /etc/init.d/keepalived status
keepalived dead but subsys locked
[root@--- we_ops_admin]# ip add
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc htb state UP qlen
link/ether :::f4:ec:b2 brd ff:ff:ff:ff:ff:ff
inet 172.16.3.190/ brd 172.16.3.255 scope global eth0
inet6 fe80:::ff:fef4:ecb2/ scope link
valid_lft forever preferred_lft forever
#VIP漂移到master2上
[root@--- we_ops_admin]# ip add
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc htb state UP qlen
link/ether :::2d::5c brd ff:ff:ff:ff:ff:ff
inet 172.16.3.189/ brd 172.16.3.255 scope global eth0
inet 172.16.3.123/ scope global secondary eth0
inet6 fe80:::ff:fe2d:965c/ scope link
valid_lft forever preferred_lft forever
2、master1重新加入集群,VIP不会重新漂移回来,造成二次波动或者脑裂现象
#重启master1上的MySQL服务
[root@--- we_ops_admin]# /etc/init.d/mysql_3309 start
Starting MySQL (Percona Server)............... SUCCESS!
#重启master1上的keepalived服务
[root@--- we_ops_admin]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
#VIP还是在master2上,且master1上并没有VIP,因为master1上设置非抢占模式,即使优先级更高
[root@--- we_ops_admin]# ip add #master1
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc htb state UP qlen
link/ether :::f4:ec:b2 brd ff:ff:ff:ff:ff:ff
inet 172.16.3.190/ brd 172.16.3.255 scope global eth0
inet6 fe80:::ff:fef4:ecb2/ scope link
valid_lft forever preferred_lft forever [root@--- we_ops_admin]# ip add master2
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc htb state UP qlen
link/ether :::2d::5c brd ff:ff:ff:ff:ff:ff
inet 172.16.3.189/ brd 172.16.3.255 scope global eth0
inet 172.16.3.123/ scope global secondary eth0
inet6 fe80:::ff:fe2d:965c/ scope link
valid_lft forever preferred_lft forever
3、master2服务宕机(如果想让VIP重新漂移回master1上,一般情况下生成环境不允许也不建议进行二次切换)
#关闭master2实例
[root@--- we_ops_admin]# /etc/init.d/mysql_3309 stop
Shutting down MySQL (Percona Server).. SUCCESS!
[root@--- we_ops_admin]# /etc/init.d/keepalived status
keepalived dead but subsys locked
#VIP已经从master2上飘走了
[root@--- we_ops_admin]# ip add
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc htb state UP qlen
link/ether :::2d::5c brd ff:ff:ff:ff:ff:ff
inet 172.16.3.189/ brd 172.16.3.255 scope global eth0
inet6 fe80:::ff:fe2d:965c/ scope link
valid_lft forever preferred_lft forever #VIP已经落盘到master1上
[root@--- we_ops_admin]# ip add
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc htb state UP qlen
link/ether :::f4:ec:b2 brd ff:ff:ff:ff:ff:ff
inet 172.16.3.190/ brd 172.16.3.255 scope global eth0
inet 172.16.3.123/ scope global secondary eth0
inet6 fe80:::ff:fef4:ecb2/ scope link
valid_lft forever preferred_lft forever
#server-id可以证明连接到master1实例
[root@--- we_ops_admin]# /opt/app/mysql_3309/bin/mysql -urepl -prepl --socket=/opt/app/mysql_3309/tmp/mysql.sock --port= --host=172.16.3.123
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.6.-68.0-log Percona Server (GPL), Release 68.0, Revision Copyright (c) - Percona LLC and/or its affiliates
Copyright (c) , , Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show global variables like '%server_id%';
+----------------+---------+
| Variable_name | Value |
+----------------+---------+
| server_id | |
| server_id_bits | |
+----------------+---------+
rows in set (0.01 sec)
上述三个测试操作,实践了VIP从master1到master2,最后再重新漂移回master1。这个切换过程中没有任何的其他问题,说明keepalived+双主的MySQL架构的健壮性还是比较强大的,且实现了服务的高可用。
Warning: Using a password on the command line interface can be insecure. #这个错误这里测试大概会报10条
Warning: Using a password on the command line interface can be insecure.
ERROR (HY000): Can't connect to MySQL server on '172.16.3.123' (111)
Warning: Using a password on the command line interface can be insecure.
ERROR (HY000): Can't connect to MySQL server on '172.16.3.123' (111)
[root@172-16-3-189 we_ops_admin]#/etc/init.d/mysql_3309 stop #master1上停止实例
Shutting down MySQL (Percona Server).. SUCCESS!
[root@172-16-3-189 we_ops_admin]#/etc/init.d/keepalived status
keepalived dead but subsys locked
[root@172-16-3-189 we_ops_admin]#ip add #vip居然还在master2上
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
link/ether 52:54:00:2d:96:5c brd ff:ff:ff:ff:ff:ff
inet 172.16.3.189/22 brd 172.16.3.255 scope global eth0
inet 172.16.3.123/22 scope global secondary eth0
inet6 fe80::5054:ff:fe2d:965c/64 scope link
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
link/ether 52:54:00:f4:ec:b2 brd ff:ff:ff:ff:ff:ff
inet 172.16.3.190/22 brd 172.16.3.255 scope global eth0
inet 172.16.3.123/22 scope global secondary eth0
inet6 fe80::5054:ff:fef4:ecb2/64 scope link
valid_lft forever preferred_lft forever
#master2上有VIP,但应用没有连接到master2上且表的行数不增长
mysql> select max(id) from test_keepalived;
+---------+
| max(id) |
+---------+
| 168 |
+---------+
1 row in set (0.00 sec)
+---------+
| max(id) |
+---------+
| 168 |
+---------+
1 row in set (0.00 sec)
mysql> select max(id) from test_keepalived;
+---------+
| max(id) |
+---------+
| 387 |
+---------+
1 row in set (0.00 sec)
+---------+
| max(id) |
+---------+
| 388 |
+---------+
1 row in set (0.00 sec)
2、master2同步被中断的问题,没有等待同步完成的机制。(VIP在maste2上时,因为master2上已经写入了数据但没来得及同步到master1上;master2实例停止后,VIP也漂移到master1,应用连接master1进行写入,但因为表设计为主键自增长,会出现ID为25已写入master2而没有同步到master1,应用连接master1写入到数据库同步到master2时报主键重复)
mysql> show slave status \G;
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.16.3.190
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: binlog.
Read_Master_Log_Pos:
Relay_Log_File: relay_bin.
Relay_Log_Pos:
Relay_Master_Log_File: binlog.
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error: Error 'Duplicate entry '' for key 'PRIMARY'' on query. Default database: 'practice'. Query: 'insert into test_keepalived values(null,1,4)'
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error: Error 'Duplicate entry '' for key 'PRIMARY'' on query. Default database: 'practice'. Query: 'insert into test_keepalived values(null,1,4)'
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 1b589d80-f450-11e7--525400f4ecb2
Master_Info_File: /opt/app/mysql_3309/logs/master.info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: ::
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position:
row in set (0.00 sec)
Keepalived+双主架构总结
keepalived+双主实践HA的更多相关文章
- MariaDB+Keepalived双主高可用配置MySQL-HA
利用keepalived构建高可用MySQL-HA,保证两台MySQL数据的一致性,然后用keepalived实现虚拟VIP,通过keepalived自带的服务监控功能来实现MySQL故障时自动切换. ...
- 企业Nginx+Keepalived双主架构案例实战
通过上一次课程的学习,我们知道Nginx+keepalived主从配置,始终有一台服务器处于空余状态,那如何更好的利用起来呢,我们需要借助Nginx+keepalived双主架构来实现,如下图通过改装 ...
- Mysql+Keepalived双主热备高可用操作记录
我们通常说的双机热备是指两台机器都在运行,但并不是两台机器都同时在提供服务.当提供服务的一台出现故障的时候,另外一台会马上自动接管并且提供服务,而且切换的时间非常短.MySQL双主复制,即互为Mast ...
- [转] Haproxy、Keepalived双主高可用负载均衡
http://blog.chinaunix.net/uid-25266990-id-3989321.html 在测试了Nginx+Keepalived的负载均衡后,也对Haproxy+Keepaliv ...
- MySQL keepalived 双主.md
MySQL keepalived 双主搭建 环境说明 系统 IP 主机名 mysql keepalived VIP CentOS 6.8 192.168.197.61 C6-node1 5.6.36 ...
- Keepalived 双主虚拟路由配置实例
Keepalived 双主虚拟路由配置实例 演示前说明: 2台centos7.2 主机:node-00,node-01 VIP1:10.1.38.19预定node-00占有 VIP2:10.1.38. ...
- mysql+keepalived 双主热备高可用
理论介绍:我们通常说的双机热备是指两台机器都在运行,但并不是两台机器都同时在提供服务.当提供服务的一台出现故障的时候,另外一台会马上自动接管并且提供服务,而且切换的时间非常短.MySQL双主复制,即互 ...
- keepalived+双主架构部署
在高可用集群环境中,keepalived使用的是VIP,利用keepalived自带的服务监控功能和自定义脚本来实现MYSQL故障时自带切换. Keepalived基于VRRP协议,虚拟冗余路由协议, ...
- 实验:keepalived双主抢占模式和非抢占模式和IPVS
内容: 一:概念.原理 二:实验过程 一.概念 一.keepalived原理及配置解析 keepalived:vrrp协议的实现 vrrp协议:virtual router redundancy ...
随机推荐
- angularjs和jquery前端发送以http请求formdata数据
formdata是比较常见的前端发送给后端的请求,不仅可以上传数据,而且同时可以上传文件. jquery使用http请求上传formdata数据的方法: var formdata = new Form ...
- 新版CSDN-markdown编辑器使用指南
本文来自CSDN官方,分markdown原文和实际显示部分,推荐开两个窗口对比浏览 Markdown部分 @[TOC](这里写自定义目录dd标题) # 欢迎使用Markdown编辑器 你好! 这是你第 ...
- springboot排除exclude
@SpringBootApplication(exclude= {DataSourceAutoConfiguration.class})
- javascript 插入DOM节点
1.使用appendChild,把一个子节点添加到父节点的最后一个子节点,.innerText插入的是内容 HTML <!-- HTML结构 --> <p id="js&q ...
- redis的过期策略都有哪些?
1.面试题 redis的过期策略都有哪些?内存淘汰机制都有哪些?手写一下LRU代码实现? 2.面试官心里分析 1)老师啊,我往redis里写的数据怎么没了? 之前有同学问过我,说我们生产环境的redi ...
- js的算法题
1.统计一个字符串中出现最多的字母 给出一个字符串,统计出现次数最多的字母.如:“wqeqwhixswiqhdxsq”,其中出现最多的是q. js算法的实现 function findMax(str) ...
- Tensor类型
Tensor类型 1.Tensor有不同的数据类型,每种类型又有CPU和GPU两种版本: 2.默认的tensor类型是FloatTensor,t.set_default_tensor_type可以修改 ...
- 重写Object类里equals方法
package com.fff; public class Pet { private String name; private int age; public Pet(String nume,int ...
- HBase MVCC 机制介绍
关键词:MVCC HBase 一致性 本文最好结合源码进行阅读 什么是MVCC ? MVCC(MultiVersionConsistencyControl , 多版本控制协议),是一种通过数据的多版本 ...
- Android系统架构及内核简介
(来源于ThinkPHP) Android是Google公司开发的基于Linux平台的开源手机操作系统,它包括操作系统.中间件.用户界面和应用程序,而且不存在任何以往阻碍移 动产业创新的专利权障碍,并 ...