lvs+keepalive实现主从效果，以及RS健康监测和tcp，udp实现非web的负载均衡

前面文章讲到了tcp和udp负载均衡，但是没有健康监测，这几天我优化了一下上次的操作。当然，我也是用的跨网段的通讯，因为线上业务主要是海外业务，所以做了iptables流量转发

IP：

lvs-master：10.60.196.184    vip10.60.196.186

lvs-buckup：10.60.196.183    vip10.60.196.186

rs1：10.60.196.164

rs2：10.60.196.165

rs3：10.60.196.168

route：42.123.110.48   内网ip：10.60.196.180   （主要做内外网转发）

lvs配置：

vim lvs.sh:

#!/bin/bash
GW=10.60.196.254
# website director vip.
SNS_VIP=10.60.196.186
SNS_RIP1=10.60.196.165
SNS_RIP2=10.60.196.168
SNS_RIP3=10.60.196.164

logger $0 called with $1
case "$1" in
start)
# set squid vip
/sbin/ipvsadm -C
/sbin/ipvsadm --set 30 5 30

ipvsadm -A -t $SNS_VIP:60000 -s rr
ipvsadm -a -t $SNS_VIP:60000 -r $SNS_RIP1:60000 -g -w 1
ipvsadm -a -t $SNS_VIP:60000 -r $SNS_RIP2:60000 -g -w 1
ipvsadm -a -t $SNS_VIP:60000 -r $SNS_RIP3:60000 -g -w 1

ipvsadm -A -u $SNS_VIP:60000 -s rr
ipvsadm -a -u $SNS_VIP:60000 -r $SNS_RIP1:60000 -g -w 1
ipvsadm -a -u $SNS_VIP:60000 -r $SNS_RIP2:60000 -g -w 1
ipvsadm -a -u $SNS_VIP:60000 -r $SNS_RIP3:60000 -g -w 1



touch /var/lock/subsys/ipvsadm >/dev/null 2>&1
;;
stop)
/sbin/ipvsadm -C
/sbin/ipvsadm -Z
# ifconfig eth3:0 down
# route del $SNS_VIP
# route del $SNS_VIP
rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1
echo "ipvsadm stoped"
;;
status)
if [ ! -e /var/lock/subsys/ipvsadm ];then
echo "ipvsadm stoped"
exit 1
else
echo "ipvsadm OK"
fi
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0

因为lvs规则这次我是直接写到ipvsadm里面，所以没有健康监测机制，因此需要写脚本来实现：

vim rs_health.sh

#!/bin/bash
#real health monitor
SNS_VIP=10.60.196.186
SNS_RIP1=10.60.196.165
SNS_RIP2=10.60.196.168
SNS_RIP3=10.60.196.164

/usr/bin/nc -v -w 1 $SNS_RIP1 -n 60000 > /dev/null
if [[ $? -ne 0 ]]; then
/sbin/ipvsadm -d -t $SNS_VIP -r $SNS_RIP1
/sbin/ipvsadm -d -u $SNS_VIP -r $SNS_RIP1
sed -i '19s/^ /#/' /root/lvs.sh                                          (这里是编辑lvs脚本，如果端口通不了就注释对应的rs条目，由于线上需求我的lvs脚本需要死循环运行，所以这里不适合用for循环写)
sed -i '26s/^ /#/' /root/lvs.sh
else
sed -i '19s/#/ /' /root/lvs.sh
sed -i '26s/#/ /' /root/lvs.sh
fi

/usr/bin/nc -v -w 1 $SNS_RIP2 -n 60000 > /dev/null
if [[ $? -ne 0 ]]; then
/sbin/ipvsadm -d -t $SNS_VIP -r $SNS_RIP2
/sbin/ipvsadm -d -u $SNS_VIP -r $SNS_RIP2
sed -i '20s/^ /#/' /root/lvs.sh
sed -i '27s/^ /#/' /root/lvs.sh
else
sed -i '20s/#/ /' /root/lvs.sh
sed -i '27s/#/ /' /root/lvs.sh
fi

/usr/bin/nc -v -w 1 $SNS_RIP3 -n 60000 > /dev/null
if [[ $? -ne 0 ]]; then
/sbin/ipvsadm -d -t $SNS_VIP -r $SNS_RIP3
/sbin/ipvsadm -d -u $SNS_VIP -r $SNS_RIP3
sed -i '21s/^ /#/' /root/lvs.sh
sed -i '28s/^ /#/' /root/lvs.sh
else
sed -i '21s/#/ /' /root/lvs.sh
sed -i '28s/#/ /' /root/lvs.sh
fi

lvs上面keepalive的配置：

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
root@localhost
}
notification_email_from admin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_instance VI_1 {
state MASTER                            （备机主要修改为buckup）
interface eth0
virtual_router_id 51
priority 100　　　　　　　　　　（备机改98）
advert_int 1
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
10.60.196.186/32 label eth0:0
}
}

由于lvs的dr模式用的是内网做的vip，所以不需要开启ip转发功能

以上关于lvs主备配置

--------------------------------------------------------------------------  主从配置一样注意修改一下彼此的master，backup和优先级

rs配置：

#!/bin/bash

#chkconfig: 2345 79 20

#description:realserver

SNS_VIP2=10.60.196.186

. /etc/rc.d/init.d/functions

case "$1" in

start)

ifconfig lo:0 $SNS_VIP1 netmask 255.255.255.255 broadcast $SNS_VIP1

/sbin/route add -host $SNS_VIP1 dev lo:0

ifconfig lo:1 $SNS_VIP2 netmask 255.255.255.255 broadcast $SNS_VIP2

/sbin/route add -host $SNS_VIP2 dev lo:1

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

echo "RealServer Start OK"

;;

stop)

ifconfig lo:0 down

route del $SNS_VIP1 >/dev/null 2>&1

ifconfig lo:1 down

route del $SNS_VIP2 >/dev/null 2>&1

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

echo "RealServer Stoped"

;;

*)

echo "Usage: $0 {start|stop}"

exit 1

esac

exit 0

route上面配置iptables转发：

-A PREROUTING -d 42.123.110.48/32 -p tcp -m tcp --dport 60000 -j DNAT --to-destination 10.60.196.186:60000
-A PREROUTING -d 42.123.110.48/32 -p udp -m udp --dport 60000 -j DNAT --to-destination 10.60.196.186:60000
-A POSTROUTING -d 10.60.196.186/32 -p tcp -m tcp --dport 60000 -j SNAT --to-source 10.60.196.180
-A POSTROUTING -d 10.60.196.186/32 -p udp -m udp --dport 60000 -j SNAT --to-source 10.60.196.180