Nginx反向代理+负载均衡简单实现（https方式）

背景：
A服务器（192.168.1.8）作为nginx代理服务器
B服务器（192.168.1.150）作为后端真实服务器

现在需要访问https://testwww.huanqiu.com请求时从A服务器上反向代理到B服务器上

这就涉及到nginx反向代理https请求的配置了~~~

                                                                                                                                             
A服务器（192.168.1.8）上的操作流程：

1）编译安装nginx
[root@opd ~]# yum install -y pcre pcre-devel openssl openssl-devel gcc
[root@opd ~]# cd /usr/loca/src
[root@src ~]# wget http://nginx.org/download/nginx-1.8.0.tar.gz
[root@src ~]# tar -zxvf nginx-1.8.0.tar.gz
[root@src ~]# cd nginx-1.8.0
#添加www用户，其中-M参数表示不添加用户家目录，-s参数表示指定shell类型

[root@nginx-1.8.0 ~]#useradd www -M -s /sbin/nologin
[root@nginx-1.8.0 ~]##vim auto/cc/gcc
#将这句注释掉 取消Debug编译模式 大概在179行
#CFLAGS="$CFLAGS -g"

#我们再配置下nginx编译参数，编译时一定要添加--with-http_ssl_module，以便让nginx支持ssl功能！
[root@nginx-1.8.0 ~]# ./configure --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_ssl_module
[root@nginx-1.8.0 ~]#make
[root@nginx-1.8.0 ~]#make install clean

2）配置nginx
[root@nginx-1.8.0 ~]# cd /usr/local/nginx/conf
[root@nginx-1.8.0 conf]# vim nginx.conf

user  nobody;
worker_processes  8;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

events {
    worker_connections  65535;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;

    log_format  main  '$http_x_forwarded_for $remote_addr $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_cookie" $host $request_time';
    sendfile       on;
    tcp_nopush     on;
    tcp_nodelay    on;
    keepalive_timeout  65;

    fastcgi_connect_timeout 3000;
    fastcgi_send_timeout 3000;
    fastcgi_read_timeout 3000;
    fastcgi_buffer_size 256k;
    fastcgi_buffers 8 256k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_intercept_errors on;

    client_header_timeout 600s;
    client_body_timeout 600s;

    client_max_body_size 100m;
    client_body_buffer_size 256k;   
 
   ## support more than 15 test environments
    server_names_hash_max_size 512;
    server_names_hash_bucket_size 128;
    gzip  on;
    gzip_min_length  1k;
    gzip_buffers     4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 9;
    gzip_types       text/plain application/x-javascript text/css application/xml text/javascript application/x-httpd-php;
    gzip_vary on;

    include vhosts/*.conf;
}

[root@nginx-1.8.0 conf]# ulimit -n 65535
[root@nginx-1.8.0 conf]# mkdir vhosts

===================================================================
接下来手动配置ssl证书
如果自己手动颁发证书的话，那么https是不被浏览器认可的，就是https上面会有一个大红叉！
===================================================================

下面是手动颁发证书的操作：
[root@linux-node1 ~]# cd /usr/local/nginx/conf/
[root@linux-node1 conf]# mkdir ssl
[root@linux-node1 conf]# cd ssl/
[root@linux-node1 ssl]# openssl genrsa -des3 -out aoshiwei.com.key 1024
Generating RSA private key, 1024 bit long modulus
................................++++++
....................................++++++
e is 65537 (0x10001)
Enter pass phrase for aoshiwei.com.key:                    #提示输入密码，比如这里我输入123456
Verifying - Enter pass phrase for aoshiwei.com.key:     #确认密码，继续输入123456

[root@linux-node1 ssl]# ls                                       #查看，已生成CSR(Certificate Signing Request)文件
aoshiwei.com.key

[root@linux-node1 ssl]# openssl req -new -key aoshiwei.com.key -out aoshiwei.com.csr
Enter pass phrase for aoshiwei.com.key:                      #输入123456
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:cn                                                         #国家
State or Province Name (full name) []:beijing                                               #省份
Locality Name (eg, city) [Default City]:beijing                                               #地区名字
Organization Name (eg, company) [Default Company Ltd]:huanqiu                 #公司名
Organizational Unit Name (eg, section) []:Technology                                     #部门
Common Name (eg, your name or your server's hostname) []:huanqiu            #CA主机名
Email Address []:wangshibo@xqshijie.cn                                                      #邮箱

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:123456                                                                   #证书请求密钥，CA读取证书的时候需要输入密码
An optional company name []:huanqiu                                                          #-公司名称，CA读取证书的时候需要输入名称

[root@linux-node1 ssl]# ls
aoshiwei.com.csr aoshiwei.com.key

[root@linux-node1 ssl]# cp aoshiwei.com.key aoshiwei.com.key.bak
[root@linux-node1 ssl]# openssl rsa -in aoshiwei.com.key.bak -out aoshiwei.com.key
Enter pass phrase for aoshiwei.com.key.bak:                            #输入123456
writing RSA key
[root@linux-node1 ssl]# openssl x509 -req -days 365 -in aoshiwei.com.csr -signkey aoshiwei.com.key -out aoshiwei.com.crt
Signature ok
subject=/C=cn/ST=beijing/L=beijing/O=huanqiu/OU=Technology/CN=huanqiu/emailAddress=wangshibo@xqshijie.cn
Getting Private key
[root@linux-node1 ssl]# ll
total 24
-rw-r--r-- 1 root root 960 Sep 12 16:01 aoshiwei.com.crt
-rw-r--r-- 1 root root 769 Sep 12 15:59 aoshiwei.com.csr
-rw-r--r-- 1 root root 887 Sep 12 16:01 aoshiwei.com.key
-rw-r--r-- 1 root root 963 Sep 12 16:01 aoshiwei.com.key.bak

然后配置nginx的反向代理：
[root@linux-node1 vhosts]# pwd
/usr/local/nginx/conf/vhosts
[root@linux-node1 vhosts]# cat test.xqshijie.com-ssl.conf
upstream 8090 {
    server 192.168.1.150:8090 max_fails=3 fail_timeout=30s;; 
}

server {
   listen 443;
   server_name testwww.huanqiu.com;
   ssl on;

### SSL log files ###
   access_log logs/ssl-access.log;
   error_log logs/ssl-error.log;

### SSL cert files ###
   ssl_certificate ssl/aoshiwei.com.crt;      #由于这个证书是自己手动颁发的，是不受信任的，访问时会有个“大叉”提示，但是不影响访问https://testwww.huanqiu.com
   ssl_certificate_key ssl/aoshiwei.com.key;   #如果是线上环境，可以购买被信任后的证书，拷贝过来使用。
   ssl_session_timeout 5m;

location / {
   proxy_pass https://8090;                                      #这个一定要是https
   proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
   proxy_set_header Host $host;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-Forwarded-Proto https;
   proxy_redirect off;
}
}

重启nginx
[root@linux-node1 ssl]# /usr/local/nginx/sbin/nginx -t
[root@linux-node1 ssl]# /usr/local/nginx/sbin/nginx -s reload

[root@linux-node1 ssl]# lsof -i:443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 15755 nobody 24u IPv4 4717921 0t0 TCP *:https (LISTEN)
nginx 15756 nobody 24u IPv4 4717921 0t0 TCP *:https (LISTEN)
nginx 15757 nobody 24u IPv4 4717921 0t0 TCP *:https (LISTEN)
nginx 15758 nobody 24u IPv4 4717921 0t0 TCP *:https (LISTEN)

A服务器要开启防火墙了，则需要在iptables里开通443端口的访问
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT

[root@linux-node1 ssl]# /etc/init.d/iptables restart

                                                                                                                                
后端真实服务器（192.168.1.150）上的nginx配置：

[root@dev-new-test1 vhosts]# cat test.xqshijie.com-ssl.conf
server {
   listen 8090;                                                                    #这里后端服务器的https没有采用默认的443端口

server_name testwww.huanqiu.com;
   root /var/www/vhosts/test.huanqiu.com/httpdocs/main/;

ssl on;
   ssl_certificate /Data/app/nginx/certificates/xqshijie.cer;          #这是后端服务器上的证书，这个是购买的被信任的证书，可以把它的证书拷贝给上面的代理机器使用
   ssl_certificate_key /Data/app/nginx/certificates/xqshijie.key;   #可以将这两个证书拷给上面192.168.1.8的/usr/loca/nginx/conf/ssl下使用，修改nginx代理配置部分的证书路径即可！

ssl_session_timeout 5m;

ssl_protocols SSLv2 SSLv3 TLSv1;
   ssl_ciphers HIGH:!aNULL:!MD5;
   ssl_prefer_server_ciphers on;

access_log /var/www/vhosts/test.huanqiu.com/logs/clickstream_ssl.log main;

location / {
   try_files $uri $uri/ @router;
   index index.php;
}

error_page 500 502 503 504 /50x.html;

location @router {
   rewrite ^.*$ /index.php last;
}

location ~ \.php$ {
  fastcgi_pass 127.0.0.1:9001;
  fastcgi_read_timeout 300;
  fastcgi_index index.php;
  fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
  #include fastcgi_params;
  include fastcgi.conf;
  fastcgi_param HTTPS on;        #这个一定要加上，否则访问https时会出现报错：The plain HTTP request was sent to HTTPS port
}
} ##end server

[root@dev-new-test1 vhosts]# lsof -i:8090
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 24373 root 170u IPv4 849747 0t0 TCP *:8090 (LISTEN)
nginx 25897 nobody 170u IPv4 849747 0t0 TCP *:8090 (LISTEN)
nginx 25898 nobody 170u IPv4 849747 0t0 TCP *:8090 (LISTEN)

最后在浏览器里访问https://testwww.huanqiu.com就能通过192.168.1.8服务器反向代理到192.168.1.150上的8090端口上了~

下面顺便附上一个测试的nginx代理配置（http和https）

[root@linux-node1 vhosts]# cat testhuanqiu.com
upstream 8802 {
   server 192.168.1.150:8802 max_fails=3 fail_timeout=30s;
}
upstream 8803 {
   server 192.168.1.150:8803 max_fails=3 fail_timeout=30s;
}
upstream 8804 {
   server 192.168.1.150:8804 max_fails=3 fail_timeout=30s;
}
upstream 8805 {
  server 192.168.1.150:8805 max_fails=3 fail_timeout=30s;
}

server {
  listen 80;
  server_name test10erp.fangfull.com;
location / {
  proxy_store off;
  proxy_redirect off;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header Host $http_host;
  proxy_pass http://8802;
}
}

server {
  listen 80;
  server_name test10www.fangfull.com;
location / {
  proxy_store off;
  proxy_redirect off;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header Host $http_host;
  proxy_pass http://8803;
}
}

server {
  listen 443;
  server_name test10fanghu.xqshijie.com;
  ssl on;

### SSL cert files ###
  ssl_certificate ssl/xqshijie.cer;
  ssl_certificate_key ssl/xqshijie.key;
  ssl_session_timeout 5m;

location / {
  proxy_pass https://8804;
  proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto https;
  proxy_redirect off;
}
}

server {
  listen 443;
  server_name test10www.xqshijie.com;
  ssl on;

### SSL cert files ###
  ssl_certificate ssl/xqshijie.cer;
  ssl_certificate_key ssl/xqshijie.key;
  ssl_session_timeout 5m;

location / {
  proxy_pass https://8805;
  proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto https;
  proxy_redirect off;
}
}

=================================================================================
上面的情况是：nginx代理层和后端服务器上都有ssl证书。
如果是nginx+tomcat+https在本机部署（即没有代理层），可以参考：https://pan.baidu.com/s/1jHPPMK2       提取密码：j7s4
=================================================================================

免费https证书可以参考 (apache和nginx都可以使用，效果很强悍！)：
https://github.com/Neilpang/acme.sh#3-install-the-issued-cert-to-apachenginx-etc
https://my.oschina.net/jianhui1980/blog/1612469
https://www.cnblogs.com/jxlwqq/p/7492274.html （推荐用这个链接，智能从apache配置中自动完成配置, 不需要指定网站根目录）

配置记录如下 (确保服务器的80和443端口在防火墙里已经提前放行):

===========================================================================================
配置http：

[root@test-web-php extra]# cat httpd-vhosts.conf
<VirtualHost *:80>
    ServerAdmin webmaster@example.com
    DocumentRoot "/www/wwwroot/www.kevin.com/web"
    ServerName 0207917c.www.kevin.com
    ServerAlias www.kevin.com
    errorDocument 404 /404.html
    ErrorLog "/www/wwwlogs/www.kevin.com-error_log"
    CustomLog "/www/wwwlogs/www.kevin.com-access_log" combined

    #DENY FILES
     <Files ~ (\.user.ini|\.htaccess|\.git|\.svn|\.project|LICENSE|README.md)$>
       Order allow,deny
       Deny from all
    </Files>

    #PHP
    <FilesMatch \.php$>
            SetHandler "proxy:unix:/tmp/php-cgi-70.sock|fcgi://localhost"
    </FilesMatch>

    #PATH
    <Directory "/www/wwwroot/www.kevin.com/web">
        SetOutputFilter DEFLATE
        Options FollowSymLinks
        AllowOverride All
        Require all granted
        DirectoryIndex index.php index.html index.htm default.php default.html default.htm
    </Directory>
</VirtualHost>

===========================================================================================
配置https:

[root@test-web-php extra]# cat httpd-ssl.conf
Listen 0.0.0.0:443
SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLHonorCipherOrder on
SSLProtocol all -SSLv3
SSLProxyProtocol all -SSLv3
SSLPassPhraseDialog  builtin
SSLSessionCache        "shmcb:/www/server/apache/logs/ssl_scache(512000)"
SSLSessionCacheTimeout  300

<VirtualHost *:443>
    ServerAdmin webmaster@example.com
    DocumentRoot "/www/wwwroot/www.kevin.com/web"
    ServerName  www.kevin.com
    ServerAlias www.kevin.com
    errorDocument 404 /404.html
    SSLEngine on
    SSLCertificateFile "/home/ubuntu/www/ssl/cert.pem"
    SSLCertificateKeyFile "/home/ubuntu/www/ssl/key.pem"
    SSLCertificateChainFile "/home/ubuntu/www/ssl/fullchain.pem"
    ErrorLog "/www/wwwlogs/www.kevin.com-error_log"
    CustomLog "/www/wwwlogs/www.kevin.com-access_log" combined

    #DENY FILES
     <Files ~ (\.user.ini|\.htaccess|\.git|\.svn|\.project|LICENSE|README.md)$>
       Order allow,deny
       Deny from all
    </Files>

    #PHP
    <FilesMatch \.php$>
            SetHandler "proxy:unix:/tmp/php-cgi-70.sock|fcgi://localhost"
    </FilesMatch>

    #PATH
    <Directory "/www/wwwroot/www.kevin.com/web">
        SetOutputFilter DEFLATE
        Options FollowSymLinks
        AllowOverride All
        Require all granted
        DirectoryIndex index.php index.html index.htm default.php default.html default.htm
    </Directory>
</VirtualHost>

===========================================================================================
配置http->https强转

[root@test-web-php extra]# cat /www/wwwroot/www.kevin.com/web/.htaccess
<IfModule mod_rewrite.c>
    Options +FollowSymlinks -Multiviews
    RewriteEngine on

    # RewriteBase /
     RewriteCond %{SERVER_PORT} !^443$
     RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [L,R]

    # if a directory or a file exists, use it directly
    RewriteCond %{REQUEST_FILENAME} !-f

    # otherwise forward it to index.php
    RewriteRule . index.php
</IfModule>
# General setting to properly handle LimeSurvey paths
# AcceptPathInfo on

===========================================================================================
证书路径：

[root@test-web-php extra]# ll /home/ubuntu/www/ssl/
total 12
-rw-r--r-- 1 root root 1903 May 23 00:53 cert.pem
-rw-r--r-- 1 root root 3551 May 23 00:53 fullchain.pem
-rw------- 1 root root 1679 May 23 00:53 key.pem