双nginx(主备、主主)反向代理tomcat实现web端负载均衡

经过以前做完的产品，受前公司几位前辈技术大拿指点，来自己动手实现并总结一下web端的负载解决方法，高手请略过，个人认知有限，请各位指正错误。

下面是结构图：

我的系统环境是Fedora22(适用readhat,centos)

2台物理机分别用于nginx的master和slaver

192.168.10.244

192.168.10.245

0.准备keepalived,nginx,

可以使用wget下载，也可直接去官网下载，这里我下载的版本是：

keepalived-1.2.19.tar.gz (官方下载地址：http://www.keepalived.org/download.html)

nginx-1.9.3.tar.gz (官方下载地址：http://nginx.org/en/download.html)

压缩包授权

 chmod  keepalived-1.2..tar.gz
 chmod  nginx-1.9..tar.gz

解压到相应的目录(随便哪个目录，注意这里不是安装目录，我的是/home/ops/tools/)

 tar -zxvf keepalived-1.2..tar.gz
 tar -zxvf nginx-1.9..tar.gz 

1.安装nginx（nginx依赖其它组件，因此需要先安装依赖，官网文档有详细说明）

进入解压后的nginx所在目录：

cd /usr/local/nginx/nginx-1.9.

检查nginx依赖是否安装：

./configure

出现：

checking for C compiler ... not found

./configure : error: C compiler cc is not found

缺少gcc依赖，安装gcc,执行：

dnf -y install gcc

安装完之后再次检查依赖是否齐全：

./configure

出现：

./configure: error :the HTTP rewrite module requires the PCRE library . 后面巴拉巴拉一片

缺少pcre库依赖，安装pcre-devel

dnf -y install pcre-devel

安装完之后再次检查依赖是否齐全命令同上，出现：

 ./configure: error: the HTTP gzip module requires the zlib library.
 You can either disable the module by using --without-http_gzip_module
 option, or install the zlib library into the system, or build the zlib library
 statically from the source with nginx by using --with-zlib=<path> option.

依次同上，再次安装zlib

dnf -y install zlib-devel

安装完,出现如下类似日志，说明nginx依赖安装成功(http://nginx.org/en/docs/configure.html，最新版本的doc文档上没有提到openssl可以不安装，原因不明，烦请知道的大哥告知一声)。

Configuration summary
+ using system PCRE library
+ OpenSSL library is not used
+ using builtin md5 code
+ sha1 library is not found
+ using system zlib library

nginx path prefix: "/usr/local/nginx"
nginx binary file: "/usr/local/nginx/sbin/nginx"
nginx configuration prefix: "/usr/local/nginx/conf"
nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
nginx pid file: "/usr/local/nginx/logs/nginx.pid"
nginx error log file: "/usr/local/nginx/logs/error.log"
nginx http access log file: "/usr/local/nginx/logs/access.log"
nginx http client request body temporary files: "client_body_temp"
nginx http proxy temporary files: "proxy_temp"
nginx http fastcgi temporary files: "fastcgi_temp"
nginx http uwsgi temporary files: "uwsgi_temp"
nginx http scgi temporary files: "scgi_temp"

OK，此时开始编译安装nginx，配置nginx的安装目录，个人习惯将其所有配置都安装在固定的目录下/usr/local/nginx,目录不存在时，安装中自动创建(官网上根据职责安装到不同的目录),分别执行:

./configure --prefix=/usr/local/nginx/
make && make install

configure的作用不知道的烦请自行查阅，安装完成后出现如下类似界面，说明安装成功，此时可以删除下载和解压的nginx源码包了.

我习惯用源码安装

官网推荐源码安装的例子有这么一段:

 ./configure
     --sbin-path=/usr/local/nginx/nginx
     --conf-path=/usr/local/nginx/nginx.conf
     --pid-path=/usr/local/nginx/nginx.pid
     --with-http_ssl_module
     --with-pcre=../pcre-8.40
     --with-zlib=../zlib-1.2.

所以我就直接拷贝下来,进入源码解压目录执行:

 ./configure --sbin-path=/usr/local/nginx/nginx --conf-path=/usr/local/nginx/nginx-1.13./nginx.conf --pid-path=/usr/local/nginx/nginx-1.13./nginx.pid --with-http_ssl_module --with-pcre=../pcre-8.40 --with-zlib=../zlib-1.2. --with-openssl=../openssl

进入nginx安装目录，启动nginx测试一下

cd /usr/local/nginx/sbin
./nginx

查看一下进程:

ps -aux | grep nginx

外部浏览器访问出现如下图，表示nginx启动成功

2.安装keepalived

进入解压后的keepalived的目录，同上先检查依赖是否安装

./configure

若出现:

configure: error:
  !!! OpenSSL is not properly installed on your system. !!!!!! Can not include OpenSSL headers files.            !!!

安装openssl依赖

dnf -y install openssl-devel

再次检查依赖，若出现如下类似说明依赖安装成功

 Keepalived configuration
 ------------------------
 Keepalived version       : 1.2.
 Compiler                 : gcc
 Compiler flags           : -g -O2
 Extra Lib                : -lssl -lcrypto -lcrypt
 Use IPVS Framework       : Yes
 IPVS sync daemon support : Yes
 IPVS use libnl           : No
 fwmark socket support    : Yes
 Use VRRP Framework       : Yes
 Use VRRP VMAC            : Yes
 SNMP support             : No
 SHA1 support             : No
 Use Debug flags          : No

OK,此时可以正式安装keepalived了，个人安装在/usr/local/keepalived目录

./configure --prefix=/usr/local/keepalived
make && make install

安装成功后进入安装后的目录，检测

cd /usr/local/keepalived/sbin
./keepalived

查看是否启动成功

ps -aux | grep keepalived

另一台机器以同样的方式安装nginx和keepalived

3.配置keepalived和nginx（测试时可以先略过这一步）

进入目录/usr/local/keepalived/sbin/,查看keepalived的启动配置参数

cd /usr/local/keepalived/sbin
./keepalived --help

注：该help参数可以在/usr/local/keepalived/etc/sysconfig/keepalived这个文件中查看）

keepalived启动时默认加载/etc/keepalived/keepalived.conf。此处个人遇到很大的坑，以为keepalived跟nginx一样安装到指定目录后会从这个安装后的目录去加载配置文件，没想到启动后日志一直停留在start...使用 ip a 查看虚拟IP也绑定不了。最后没办法只好在etc下新建keepalived目录，把keepalived.conf拷过去才解决。

mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf  /etc/keepalived

我的2台nginx配置文件nginx.conf部分配置片段如下:(这里我在118的机器上起了4个tomcat实例，nginx不配负载方式默认是轮询),用作测试

 # blentle add
 upstream www.test.dev {
         server 192.168.10.118:;
         server 192.168.10.118:;
         server 192.168.10.118:;
         server 192.168.10.118:;
 }

 server {
         listen ;
         server_name 127.0.0.1;
         location / {
         index index.php index.html index.jsp;
         proxy_pass http://www.test.dev;
         proxy_set_header   X-Real-IP  $remote_addr;
         proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
         }

 }

4.配置keepalived，实现nginx负载

（1）主从(master-slaver)

所谓主从即master的nginx正常提供服务时，永远轮不到slaver的nginx来提供服务。只有当master挂掉时，slaver的nginx才接管虚拟IP，提供正常服务。貌似有点浪费服务器。

（a）配置master(192.168.10.244)

vi /etc/keepalived/keepalived.conf

将以下内容替换keepalived.conf里面的内容

 ! Configuration File for keepalived
 global_defs {
     notification_email {
         admin@milipp.com
     }
     notification_email_from dev@milipp.com
     smtp_server mail.milipp.com
     smtp_connect_timeout
     router_id LVS_DEVEL
 }
11 vrrp_script chk_nginx {
12     script "/home/ops/scripts/monitor-nginx.sh"  #这个即使监控nginx宕机的脚本实现自动切换
13     interval 2
14     weight -5
15     fall 3
16     rise 2
17 }
 vrrp_instance VI_1 {
     state MASTER
     interface eth0  #主nginx ip对应的网卡名,使用ifconfig查看
     mcast_src_ip 192.168.10.244 #主nginx的ip
     virtual_router_id    #虚拟路由id，主从必须一致
     priority          #优先级，主比从大
     advert_int
     authentication {
         auth_type PASS
         auth_pass
     }
     virtual_ipaddress {
         192.168.10.246  #虚拟ip,主从必须一致，可以陪配置多个
     }
     track_script {
        chk_nginx  #上面配的监控nginx的脚本
     }
 }

(b).配置slaver(192.168.10.245)

vi /etc/keepalived/keepalived.conf

将以下内容替换keepalived.conf里面的内容

 ! Configuration File for keepalived
 global_defs {
     notification_email {
         renhuan@milipp.com
     }
     notification_email_from xuyongming@milipp.com
     smtp_server mail.example.com
     smtp_connect_timeout
     router_id LVS_DEVEL
 }
 vrrp_script chk_nginx {
     script  "/home/ops/scripts/monitor_nginx.sh"
     interval
     weight -
     fall
     rise
 }
 vrrp_instance VI_1 {
     state BACKUP
     interface enp3s0
     mcast_src_ip 192.168.10.245
     virtual_router_id
     priority
     advert_int
     authentication {
         auth_type PASS
         auth_pass
     }
     virtual_ipaddress {
         192.168.10.246
     }
     track_script {
        chk_nginx
     }
 }

监控脚本/home/ops/scripts/monitor_nginx.sh的内容:

 #!/bin/bash
 counter=$(ps -C nginx --no-heading|wc -l)
 if [ "${counter}" = "" ]; then
     /usr/local/nginx/sbin/nginx
     sleep
     counter=$(ps -C nginx --no-heading|wc -l)
     if [ "${counter}" = "" ]; then
         /usr/local/keepalived/sbin/keepalived stop
     fi
 fi

即当nginx挂掉后，停掉keepalived

测试:

分别启动两台nginx和两台keepalived，使用虚拟ip 192.168.10.246访问。

在244上执行：

ip a

得到结果：

虚拟ip绑定到244上，以同样的方式查看245，结果245没有绑定虚拟ip

此时停掉245的keepalived或者nginx,使用虚拟ip 246访问，没有问题，查看绑定情况,发现245绑定了虚拟ip。至此测试成功。

（2）主主(master master)

TODO: