keepalived+LVS实现网站负载均衡和HA

　　

　　如上图所示，102和103是内网nginx服务器，100和101是边界LB，clinet是1，这个实验是为了实现在LB上虚拟出一个VIP，client通过访问该VIP，来动态负载到两台内网nginx服务器上面来。流量的来回，都需要经过边界LB。

　　我们首先分析一下，这种模式是一种NAT模式，流量的来回都要经过边界LB。回忆一下NAT的工作模式，LB在接收到client的数据请求后，根据自身负载均衡算法，将目标IP地址修改为内网RS地址，然后从内网口扔出去，内网服务器收到这个数据请求之后，目标MAC和IP都是自己，源地址是client，这个时候就会处理该数据包，返回，返回的数据包中，源地址是自己，目标地址是client，在经过LB的时候，会被转换，源地址为VIP地址，目标地址为client地址，然后client就会收到这个回包。

　　这个和物理硬件LB的工作模式中，透明模式相同，透明模式也就是NAT模式。而物理硬件中的反向代理模式，和LVS中的tunnel模式相似，在接收到client的数据请求后，会将整个数据包进行改写，源地址为LB的内网地址，目标地址为某一个RS地址，然后数据包返回的时候，也就会再次进行转换。这种反向代理模式中，服务器正常情况下无法知道client的源，只知道是自己的LB访问了自己，可以通过配置LB，添加字段，来讲该client的地址一并发送给RS。

　　物理硬件LB中的三角模式，也就是LVS的DR模式。需要在服务器上配置lo接口。

　　回到环境中。这个地方采用的是NAT模式，也就是说流量必须要经过LB，最简单的做法，就是将两台NGINX服务器的网关，配置在LB上，而且是以VIP的形式存在，对外，LB也拿出一个VIP，作为client访问的入口，也就是说，LB上，有两个VRRP INSTANCE，而且要保持状态一致。这样才能实现高可用HA。

　　下面开始做配置了，主要是在两台LB上配置keepalived，keepalived是默认安装的，配置文件存放在/etc/keepalived/目录下，启动脚本为/etc/init.d/keepalived，定义100为主，101为备。下面是两台设备的keepalived配置文件。

! Configuration File for keepalived
! The global defines,including notification server and target,router-id and others
global_defs {
    notification_email {
        admin@domain.com
        @domain.com
    }
    notification_email_from keepalived@domain.com
    smtp_server 127.0.0.1
    smtp_connect_timeout
    router_id LB_100
}
! The vrrp sync group,any vrrp instance in the vrrp sync group while keep the same status,
!       no matter the priority ,for example ,VI_1 is master ,VI_2 is backup,the sync group status is backup
vrrp_sync_group VG1 {
    group {
        VI_To_client
        VI_To_server
    }
}
! The VRRP instance for client to connect,clients connect to the VIPs
vrrp_instance VI_To_client {
    state MASTER
    interface eth0
    virtual_router_id
    priority
    nopreempt
    advert_int
    authentication {
        auth_type PASS
        auth_pass
    }
    virtual_ipaddress {
        192.168.245.200
    }
}
! The VRRP instance for real servers,real servers` default gateway
vrrp_instance VI_To_server {
    state MASTER
    interface eth1
    virtual_router_id
    priority
    nopreempt
    advert_int
    authentication {
        auth_type PASS
        auth_pass
    }
    virtual_ipaddress {
        10.0.0.254
    }
}
! The LB configuration,including VS and RS,scheduling argu,and other options,health check
virtual_server 192.168.245.200  {
    delay_loop
    lb_algo rr
    lb_kind NAT
    persistence_timeout
    protocol TCP

    real_server 10.0.0.102  {
        weight
        TCP_CHECK {
            connect_timeout
        }
    }

    real_server 10.0.0.103  {
        weight
        TCP_CHECK {
            connect_timeout
        }
    }
}

　　在100的配置文件中，主要是配置了一个vrrp_sync_group，里面有两个VRRP INSTANCE，分别是面向client和面向RS的，两个VRRP INSTANCE的vrrp-_id是不同的，但是其他属性值都是ixiangtong的。另外，在keepalived的LB配置中，有对VS和RS的配置，同时对RS使用了health check，检查方法很简单，就是TCP的connect，当能够访问到时，将该RS动态的加入到ipvs中，如果访问不到，就从ipvs中移除。这个现象可以通过查看日志看到。

　　slave的配置文件和master的配置文件大同小异，除了必要的字段不同，其他都是相同的。下面看看keepalived的日志信息。

Jul  :: CentOS_101 Keepalived[]: Starting Keepalived v1.2.13 (/,)
Jul  :: CentOS_101 Keepalived[]: Starting Healthcheck child process, pid=
Jul  :: CentOS_101 Keepalived[]: Starting VRRP child process, pid=
Jul  :: CentOS_101 Keepalived_vrrp[]: Netlink reflector reports IP 192.168.245.101 added
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Netlink reflector reports IP 192.168.245.101 added
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Netlink reflector reports IP 10.0.0.101 added
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Netlink reflector reports IP fe80::20c:29ff:fe88: added
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Netlink reflector reports IP fe80::20c:29ff:fe88: added
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Registering Kernel netlink reflector
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Registering Kernel netlink command channel
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Opening file '/etc/keepalived/keepalived.conf'.
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Configuration is using :  Bytes
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Using LinkWatch kernel netlink reflector...
Jul  :: CentOS_101 Keepalived_vrrp[]: Netlink reflector reports IP 10.0.0.101 added
Jul  :: CentOS_101 Keepalived_vrrp[]: Netlink reflector reports IP fe80::20c:29ff:fe88: added
Jul  :: CentOS_101 Keepalived_vrrp[]: Netlink reflector reports IP fe80::20c:29ff:fe88: added
Jul  :: CentOS_101 Keepalived_vrrp[]: Registering Kernel netlink reflector
Jul  :: CentOS_101 Keepalived_vrrp[]: Registering Kernel netlink command channel
Jul  :: CentOS_101 Keepalived_vrrp[]: Registering gratuitous ARP shared channel
Jul  :: CentOS_101 Keepalived_vrrp[]: Opening file '/etc/keepalived/keepalived.conf'.
Jul  :: CentOS_101 Keepalived_vrrp[]: Configuration is using :  Bytes
Jul  :: CentOS_101 Keepalived_vrrp[]: Using LinkWatch kernel netlink reflector...
Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Activating healthchecker for service [10.0.0.102]:80
Jul 15 13:57:01 CentOS_101 Keepalived_healthcheckers[26015]: Activating healthchecker for service [10.0.0.103]:80
Jul  :: CentOS_101 Keepalived_vrrp[]: VRRP_Instance(VI_To_client) Entering BACKUP STATE
Jul  :: CentOS_101 Keepalived_vrrp[]: VRRP_Instance(VI_To_server) Entering BACKUP STATE
Jul  :: CentOS_101 Keepalived_vrrp[]: VRRP sockpool: [ifindex(), proto(), unicast(), fd(,)]
Jul  :: CentOS_101 Keepalived_vrrp[]: VRRP sockpool: [ifindex(), proto(), unicast(), fd(,)]

　　这个是启动日志，在启动的时候，就对两个RS就开启了健康检查。

Jul  :: CentOS_101 Keepalived_healthcheckers[]: TCP connection to [10.0.0.102]: failed !!!
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Removing service [10.0.0.102]: from VS [192.168.245.200]:
Jul  :: CentOS_101 Keepalived_healthcheckers[]: TCP connection to [10.0.0.102]: success.
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Adding service [10.0.0.102]: to VS [192.168.245.200]:
Jul  :: CentOS_101 Keepalived_healthcheckers[]: TCP connection to [10.0.0.103]: failed !!!
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Removing service [10.0.0.103]: from VS [192.168.245.200]:
Jul  :: CentOS_101 Keepalived_healthcheckers[]: TCP connection to [10.0.0.103]: success.
Jul  :: CentOS_101 Keepalived_healthcheckers[]: Adding service [10.0.0.103]: to VS [192.168.245.200]:

　　这个是测试RS故障时候，keepalived的日志。日志写的很清楚，当主机故障时，就从VS中移除，同时可以看到ipvsadm的状态。

[root@CentOS_100 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2. (size=)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.245.200: rr persistent
  -> 10.0.0.102:                Masq
[root@CentOS_100 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2. (size=)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.245.200: rr persistent
  -> 10.0.0.102:                Masq
  -> 10.0.0.103:                Masq
[root@CentOS_100 keepalived]# ipvsadm -L -n

　　client在访问的时候，只需要访问200这个虚拟地址就好了。如果不想LB作为RS的网关，可以直接在RS上写一条路由即可。

　　在NAT模式中，需要在LB上开启转发，先查看sysctl -p查看是否开启了转发，如果没有开启，需要在/etc/sysctl.conf中修改net.ipv4.ip_forward为1，默认为0，如果为0，数据包在经过LB的时候，是不会被转发出去的。这个时候的LB，作为的是一个路由器，用linux作为路由器，是需要开启这个属性的。