Nginx+Keepalived双主轮询负载均衡

双主模式使用两个VIP，前段有2台服务器，互为主从，两台服务器同时工作，不存在资源浪费情况。同时在前端的DNS服务器对网站做多条A记录，实现了Nginx的负载均衡，当一台服务器故障时候，资源会转移到另一台服务器，继续提供服务，在大型的网站中多数都使用此种架构。在此使用主主模式配置Nginx+keepalived的高可用性。

两台Nginx，两台Web，前端DNS由运营商提供。

IP地址

Nginx1:10.10.10.11

Nginx2:10.10.10.12

VIP1:10.10.10.21

VIP2:10.10.10.22

Real1:10.10.10.13

Real2:10.10.10.14

1，在2台Web主机上部署环境，安装Nginx+PHP+MySQL,参考我前面的文章

2，分别在二台Nginx负载均衡器上安装Nginx，配置

安装GCC编译器等工具：

yum install -y gcc gcc-c++ autoconf automake libtool make openssl openssl-devel

安装Nginx:

wget http://exim.mirror.fr/pcre/pcre-8.38.tar.gz

tar -zxvf pcre-8.38.tar.gz

cd pcre-8.38

./configure

make && make install

wget http://zlib.net/zlib-1.2.8.tar.gz

tar -zxvf zlib-1.2.8.tar.gz

cd zlib-1.2.8

./configure

make && make install

wget http://nginx.org/download/nginx-1.6.3.tar.gz

tar -zxvf nginx-1.6.3.tar.gz

cd nginx-1.6.3/

./configure --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --pid-path=/usr/local/nginx/logs/nginx.pid --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module

make && make install

Nginx.conf配置文件，二个nginx负载均衡器的文件一样

user  nobody;

worker_processes  1;

error_log  /usr/local/nginx/logs/error.log notice;

pid        /usr/local/nginx/logs/nginx.pid;

worker_rlimit_nofile 51200;

events {

use epoll;

worker_connections  51200;

}

http {

include       mime.types;

default_type  application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

access_log  logs/access.log  main;

server_names_hash_bucket_size 128;

client_header_buffer_size 32k;

large_client_header_buffers 4 32k;

client_max_body_size 8m;

sendfile        on;

tcp_nopush     on;

server_tokens off;

keepalive_timeout  60;

fastcgi_connect_timeout 300;

fastcgi_send_timeout 300;

fastcgi_read_timeout 300;

fastcgi_buffer_size 64k;

fastcgi_buffers 4 64k;

fastcgi_busy_buffers_size 128k;

fastcgi_temp_file_write_size 128k;

gzip  on;

upstream backend

{

server 10.10.10.13;

server 10.10.10.14;

}

server {

listen       80;

server_name  10.10.10.21;  #Nginx2改为10.10.10.22

location / {

root   html;

index  index.php index.html index.htm;

proxy_redirect off;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

#后端的Web服务器可以通过X-Forwarded-For获取用户真实IP

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_pass http://backend;

}

error_page   500 502 503 504  /50x.html;

location = /50x.html {

root   html;

}

location /nginx_status {

stub_status on;

auth_basic "NginxStatus";

auth_basic_user_file /usr/local/nginx/htpasswd;

#allow 127.0.0.1;

#deny all;

}

location ~* \.(ini|docx|txt|doc|pdf)$ {

#禁止访问文档性文件

root /usr/share/nginx/html;

deny all;

}

location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|js|html|htm|css)$ {

root /home/image;

proxy_store on;

proxy_store_access user:rw group:rw all:rw;

proxy_temp_path /home/image;

if ( !-e $request_filename) {

proxy_pass  http://backend;

}

}

}

}

配置完成启动服务

[root@hd1 sbin]# ./nginx

./nginx: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory

cd /lib

[root@hd1 lib]# ls

[root@hd1 lib]# ls *pcre*

libpcre.so.0  libpcre.so.0.0.1

[root@hd1 lib]# ln -s /lib/libpcre.so.0.0.1 /lib/libpcre.so.1

[root@hd1 lib]#

[root@hd1 sbin]# ./nginx

在二台Nginx上安装及配置keepalived：

wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gz

tar -zxvf keepalived-1.2.15.tar.gz

cd keepalived-1.2.15

./configure --sysconf=/etc/  --with-kernel-dir=/usr/src/kernels/2.6.32-358.el6.i686

Keepalived configuration

------------------------

Keepalived version       : 1.2.15

Compiler                 : gcc

Compiler flags           : -g -O2

Extra Lib                : -lssl -lcrypto -lcrypt

Use IPVS Framework       : Yes

IPVS sync daemon support : Yes

IPVS use libnl           : No

fwmark socket support    : Yes

Use VRRP Framework       : Yes

Use VRRP VMAC            : Yes

SNMP support             : No

SHA1 support             : No

Use Debug flags          : No

make && make install

ln -s /usr/local/sbin/keepalived  /sbin/

#这一步很重要，不执行ln -s会报错“Starting keepalived: /bin/bash: keepalived: command not found”

service keepalived start

二台Nginx上keepalived.conf配置如下，配置完成后分别service keepalived start启动，检测keepalived配置是否成功

Nginx1：

global_defs {

notification_email {

test@163.com

}

notification_email_from keepalived@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id NGINX_VIP1

}

vrrp_script chk_http_port {

script "/usr/local/src/check_nginx_pid.sh"

interval 2                           #（检测脚本执行的间隔）

weight 2

}

vrrp_instance VI_1 {

state MASTER

interface eth1

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

chk_http_port            #（调用检测脚本）

}

virtual_ipaddress {

10.10.10.21/24 broadcast 10.10.10.255 dev eth1 label eth1:1

}

}

vrrp_instance VI_2 {

state BACKUP

interface eth1

virtual_router_id 52

priority 99

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

chk_http_port            #（调用检测脚本）

}

virtual_ipaddress {

10.10.10.22/24 broadcast 10.10.10.255 dev eth1 label eth1:2

}

}

Nginx2：

global_defs {

notification_email {

test@163.com

}

notification_email_from keepalived@localhost

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id NGINX_VIP2

}

vrrp_script chk_http_port {

script "/usr/local/src/check_nginx_pid.sh"

interval 2                           #（检测脚本执行的间隔）

weight 2

}

vrrp_instance VI_1 {

state BACKUP

interface eth1

virtual_router_id 51

priority 99

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

chk_http_port            #（调用检测脚本）

}

virtual_ipaddress {

10.10.10.21/24 broadcast 10.10.10.255 dev eth1 label eth1:1

}

}

vrrp_instance VI_2 {

state MASTER

interface eth1

virtual_router_id 52

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

chk_http_port            #（调用检测脚本）

}

virtual_ipaddress {

10.10.10.22/24 broadcast 10.10.10.255 dev eth1 label eth1:2

}

}

以下是针对nginx状态进行检测的脚本

vim  /usr/local/src/check_nginx_pid.sh

#!/bin/bash

A=`ps -C nginx --no-header |wc -l`

if [ $A -eq 0 ];then

/usr/local/nginx/sbin/nginx

if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then

killall keepalived

fi

fi

脚本加上可执行权限

chmod +x /usr/local/keepalived/sbin/check_nginx.sh

服务开启后网卡状态

[root@hd2 keepalived-1.2.15]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000

link/ether 00:50:56:22:04:b1 brd ff:ff:ff:ff:ff:ff

inet 10.10.10.12/24 brd 10.10.10.255 scope global eth1

inet 10.10.10.22/24 brd 10.10.10.255 scope global secondary eth1:2

inet6 fe80::250:56ff:fe22:4b1/64 scope link

valid_lft forever preferred_lft forever

3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

link/ether be:88:be:d6:81:a6 brd ff:ff:ff:ff:ff:ff

[root@hd2 keepalived-1.2.15]#

====================================测试============================

测试web主节点服务down掉之后，备用节点服务是否能正常运行，kill -9 xxxxx，web仍然能够访问

模拟keepalived节点出现故障，nginx服务器是否能自动转移

[root@hd1 sbin]# service keepalived stop

Stopping keepalived: [  OK  ]

结果访问vip 21无法访问，但是22仍然能够正常提供服务