web架构-nginx负载均衡

nginx的负载均衡

Nginx 是一个广泛使用的反向代理服务器，能够高效地实现负载均衡。负载均衡的核心作用是将来自客户端的请求分发到多个后端服务器上，从而平衡每台服务器的压力。通过Nginx，我们可以实现多种负载均衡算法，如轮询、IP哈希等。

vi /etc/nginx/nginx.conf 插入http的下一行
    upstream app {
        server 172.16.162.215:80;
        server 172.16.162.216:80;
    }
  server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    location / {
            proxy_pass http://app;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
    }
}

在上述配置中，proxy_pass 指令将所有匹配到根路径的请求代理到后端定义的 upstream 服务器池。proxy_set_header 用于设置转发过程中传递给后端服务器的请求头信息，保证客户端的真实IP等信息可以正确地传递到后端服务器，确保后端服务器在日志记录和安全审计时能获取到完整的客户端信息。

两个不同的主机进行负载

[root@iZbp1fry44ac0aglan54biZ ~]# hostnamectl set-hostname program1
[root@iZbp1fry44ac0aglan54biZ ~]# bash
[root@program1 ~]# echo my id is 1 >  index.html
[root@program1 ~]# ls
index.html
[root@program1 ~]# python3 -m http.server  80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...

[root@iZbp11cxs557bjnz6el8roZ ~]# hostnamectl set-hostname program2
[root@iZbp11cxs557bjnz6el8roZ ~]# bash
[root@program2 ~]#  echo my id is 2 >  index.html
[root@program2 ~]# ls
index.html
[root@program2 ~]# python3 -m http.server 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...

成功进行负载均衡

[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 1
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 2
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 1
[root@iZbp11cxs557bjnz6el8roZ nfs]# curl 47.97.245.17
my id is 2
[root@iZbp11cxs557bjnz6el8roZ nfs]#

nginx的基础负载均衡，在此就配置完成了。

真正的程序不止只有一个前端页面，有数据库，有php，有静态网页等等

简单程序的基础架构

在现代分布式系统中，使用Docker容器来部署应用已经成为了一种常见的做法。Docker容器可以将应用及其依赖环境打包成一个独立的单元，确保无论在何种操作系统环境中都能一致运行。容器化的优点在于部署快速、资源隔离好，并且支持轻量级的水平扩展。不需要虚拟操作系统硬件资源

配置docker

# step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step 3
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
# Step 4: 更新并安装Docker-CE
sudo yum makecache fast
sudo yum -y install docker-ce
# Step 4: 开启Docker服务
sudo service docker start

[root@iZbp1fry44ac0aglan54biZ ~]# vi /etc/docker/daemon.json
{
  "registry-mirrors": [
    "https://dockerhub.icu",
    "https://docker.registry.cyou",
    "https://docker-cf.registry.cyou",
    "https://dockercf.jsdelivr.fyi",
    "https://docker.jsdelivr.fyi",
    "https://dockertest.jsdelivr.fyi",
    "https://mirror.aliyuncs.com",
    "https://dockerproxy.com",
    "https://mirror.baidubce.com",
    "https://docker.m.daocloud.io",
    "https://docker.nju.edu.cn",
    "https://docker.mirrors.sjtug.sjtu.edu.cn",
    "https://docker.mirrors.ustc.edu.cn",
    "https://mirror.iscas.ac.cn",
    "https://docker.rainbond.cc"
  ]
}
[root@iZbp1fry44ac0aglan54biZ ~]# systemctl restart docker

配置数据库

docker run -d \
  --name mariadb \
  -e MYSQL_ROOT_PASSWORD=Supermao666 \
  -e MYSQL_DATABASE=nextcloud \
  -p 3306:3306 \
  -v /opt/db:/var/lib/mysql \
  mysql:5.7

4f8398c27b1b2104aae735d839f4f07cb39c44dd70370dad03275953d54dd53c
[root@program1 ~]# docker ps
CONTAINER ID   IMAGE       COMMAND                  CREATED         STATUS         PORTS                                                  NAMES
4f8398c27b1b   mysql:5.7   "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mariadb

局域网内网连接测试
[root@program1 ~]# mysql -uroot -pSupermao666 -h172.16.162.215
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.44 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| nextcloud          |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

MySQL [(none)]> Bye

配置 nextcloud程序

docker网络

docker run -d \
  --name nextcloud1 \
  -e MYSQL_PASSWORD=Supermao666 \
  -e MYSQL_DATABASE=nextcloud \
  -v /opt/nextcloud1:/var/www/html \
  -v /opt/nextdata:/var/www/html/data \
  -p 80:80 \
  --network bridge \
  nextcloud:latest

[root@program1 ~]# docker ps
CONTAINER ID   IMAGE              COMMAND                  CREATED              STATUS              PORTS                                                  NAMES
69cb522262fa   nextcloud:latest   "/entrypoint.sh apac…"   About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, :::80->80/tcp                      nextcloud1
4f8398c27b1b   mysql:5.7          "docker-entrypoint.s…"   6 minutes ago        Up 6 minutes        0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mariadb
#通过公网ip来访问初始化程序
[root@program1 ~]# curl ip.sb
121.43.116.224
[root@program1 ~]#

上传文件成功，完成简单程序部署

用户访问数量变多，数据库，云主机逐渐满载

使用redis中间件来缓解mysql的压力，使用缓存而不是每次都去查询数据库。

拉大云主机的规格，或者横向拓展云主机数量，以达到负载均衡。这里使用横向拓展

使用共享nfs存储来保证，所有横向拓展的云主机，配置文件和数据保持一致。（倘若不一致等于没拓展，所有云主机各自为战）

nfs配置

[root@iZbp11cxs557bjnz6el8roZ ~]# mkdir  /nfs
[root@iZbp11cxs557bjnz6el8roZ /]# chmod -R  777 nfs/
[root@iZbp11cxs557bjnz6el8roZ /]# sudo vi /etc/exports
[root@iZbp11cxs557bjnz6el8roZ /]# cat /etc/exports
/nfs 172.16.162.0/24(rw,sync,no_root_squash,no_subtree_check)
## 局域网内部访问 sync：同步写入到磁盘，保证数据安全。
## no_root_squash：允许远程 root 用户对 NFS 共享的文件拥有 root 权限（可以根据安全需求调整）。
## no_subtree_check：禁用子树检查，能提高性能。
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl start nfs
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@iZbp11cxs557bjnz6el8roZ /]#
[root@iZbp11cxs557bjnz6el8roZ /]# sudo vi /etc/exports
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl start nfs
[root@iZbp11cxs557bjnz6el8roZ /]# sudo systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@iZbp11cxs557bjnz6el8roZ nfs]# touch flag
[root@iZbp11cxs557bjnz6el8roZ nfs]# ls
flag
[root@iZbp11cxs557bjnz6el8roZ nfs]# pwd
/nfs
[root@iZbp11cxs557bjnz6el8roZ nfs]#

nfs验证测试

[root@program1 ~]# cd /mnt/
[root@program1 mnt]# ls
[root@program1 mnt]# sudo mount -t nfs 172.16.162.217:/nfs /mnt/
[root@program1 mnt]# ls
[root@program1 mnt]# cd ..
[root@program1 /]# cd -
/mnt
[root@program1 mnt]# ls
flag
[root@program1 mnt]# df -hT
Filesystem          Type      Size  Used Avail Use% Mounted on
devtmpfs            devtmpfs  1.8G     0  1.8G   0% /dev
tmpfs               tmpfs     1.8G     0  1.8G   0% /dev/shm
tmpfs               tmpfs     1.8G  580K  1.8G   1% /run
tmpfs               tmpfs     1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/vda1           ext4       40G  6.1G   32G  17% /
tmpfs               tmpfs     365M     0  365M   0% /run/user/0
overlay             overlay    40G  6.1G   32G  17% /var/lib/docker/overlay2/e3c8da8d16909c43dd674566d7effbd34ec017300453cedacc82d6e72b809056/merged
172.16.162.217:/nfs nfs4       40G  2.6G   35G   7% /mnt
[root@program1 mnt]# 

删除之前的nextcloud 再次启动nextcloud

将nextcloud的数据盘 和 配置文件在 mnt共享磁盘里

[root@program1 mnt]# docker run -d \
   --name nextcloud21 \
   -e MYSQL_PASSWORD=Supermao666 \
   -e MYSQL_DATABASE=nextcloud \
   -v /mnt/nextcloud1:/var/www/html \
   -v /mnt/nextdata:/var/www/html/data \
   -p 80:80 \
   --network bridge \
   nextcloud:latest
ec3e2f63df4a71383be910d97a021a402ae9477eb7acefdfe9d166f61acc2c5e
[root@program1 mnt]# docker ps
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
ec3e2f63df4a   nextcloud:latest   "/entrypoint.sh apac…"   3 seconds ago    Up 2 seconds    0.0.0.0:80->80/tcp, :::80->80/tcp                      nextcloud21
4f8398c27b1b   mysql:5.7          "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mariadb
[root@program1 mnt]# du -sh
5.9M	.
[root@program1 mnt]# docker logs ec3e2f63df4a
Initializing nextcloud 29.0.6.1 ...
[root@program1 mnt]# 

[root@program1 mnt]# du -sh
29M	.
[root@program1 mnt]#
等待写入共享磁盘

## 配置完成
[root@program1 mnt]# du -sh
684M	.
[root@program1 mnt]# docker logs ec3e2f63df4a
Initializing nextcloud 29.0.6.1 ...
New nextcloud instance
Next step: Access your instance to finish the web-based installation!
Hint: You can specify NEXTCLOUD_ADMIN_USER and NEXTCLOUD_ADMIN_PASSWORD and the database variables _prior to first launch_ to fully automate initial installation.
Initializing finished
=> Searching for scripts (*.sh) to run, located in the folder: /docker-entrypoint-hooks.d/before-starting
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
[Tue Sep 10 14:47:18.756670 2024] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.62 (Debian) PHP/8.2.23 configured -- resuming normal operations
[Tue Sep 10 14:47:18.756745 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
27.17.161.6 - - [10/Sep/2024:14:47:37 +0000] "POST /apps/text/session/20/sync HTTP/1.1" 404 655 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0"
27.17.161.6 - - [10/Sep/2024:14:47:37 +0000] "POST /apps/text/session/20/push HTTP/1.1" 404 655 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0"
27.17.161.6 - - [10/Sep/2024:14:47:37 +0000] "GET /apps/files/api/v1/stats HTTP/1.1" 404 655 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0"
[root@program1 mnt]#
[root@program1 mnt]# curl ip.sb
121.43.116.224

删除之前的数据库
[root@program1 mnt]# docker ps
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
ec3e2f63df4a   nextcloud:latest   "/entrypoint.sh apac…"   9 minutes ago    Up 9 minutes    0.0.0.0:80->80/tcp, :::80->80/tcp                      nextcloud21
4f8398c27b1b   mysql:5.7          "docker-entrypoint.s…"   38 minutes ago   Up 38 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mariadb
[root@program1 mnt]# docker rm -f 4f8398c27b1b
4f8398c27b1b
[root@program1 mnt]# rm -rf /opt/db/
[root@program1 mnt]#
之前的数据库已经被写入supermao
重启数据库
[root@program1 mnt]# docker run -d   --name mariadb   -e MYSQL_ROOT_PASSWORD=Supermao666   -e MYSQL_DATABASE=nextcloud   -p 3306:3306   -v /opt/db:/var/lib/mysql   mysql:5.7

再次进入公网ip进行访问然后初始化

云主机program2配置redis

在高并发的Web系统中，频繁的数据库查询会导致性能瓶颈。为了解决这一问题，可以引入Redis作为缓存系统，减少直接对数据库的访问。Redis 是一个基于内存的键值数据库，能够非常快速地存取数据，特别适合用来缓存一些频繁查询的数据。

[root@program2 ~]# docker run -d --name redis-container -p 6379:6379 redis
545f756f22f2c768c44891aa256708699ec199ad5a4ae0788727c506b4111985
[root@program2 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                                       NAMES
545f756f22f2   redis     "docker-entrypoint.s…"   5 seconds ago   Up 4 seconds   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   redis-container
## 测试redis键值数据库
[root@program2 ~]# redis-cli -h 172.16.162.216 -p 6379
172.16.162.216:6379> set mykey "Hello Redis"
OK
172.16.162.216:6379> get mykey
"Hello Redis"
172.16.162.216:6379>

nextcloud与redis的对接

redis存在与program2的上面，ip为172.16.162.216
首先监控redis
[root@program1 ~]# redis-cli  -h 172.16.162.216 MONITOR
OK
更新nextcloud的配置文件，会自动同步进docker容器
源文件
<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),

更改后
<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\OC\Memcache\Redis',
  'filelocking.enabled' => 'true',
  'redis' => array(
    'host' => '172.16.162.216',
    'port' => '6379',
  ),
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),

提高安全性，只允许来自47.97.245.17来访问nextcloud实例。防止劫持DNS记录或伪造请求
  'trusted_domains' =>
  array (
    0 => '47.97.245.17',   #第一部分的负载均衡公网地址
  ),

若redis一直刷新则是命中缓存，redis配置成功
[root@program1 ~]# redis-cli  -h 172.16.162.216 MONITOR
OK
1725980850.548579 [0 172.16.162.215:51728] "TTL" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/b947cec9e58799d4586628341ec933a4"
1725980850.548951 [0 172.16.162.215:51728] "INCRBY" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/b947cec9e58799d4586628341ec933a4" "1"
1725980850.549195 [0 172.16.162.215:51728] "EXPIRE" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/b947cec9e58799d4586628341ec933a4" "3600"
1725980850.550609 [0 172.16.162.215:51728] "TTL" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/c5e0b4791fd3a7d14e54b84edeb73034"
1725980850.550870 [0 172.16.162.215:51728] "INCRBY" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/c5e0b4791fd3a7d14e54b84edeb73034" "1"
1725980850.551147 [0 172.16.162.215:51728] "EXPIRE" "b5f1517db060bd92ddc8106a7d81bbcf/lockfiles/c5e0b4791fd3a7d14e54b84edeb73034" "3600"
1725980850.552536 [0 172.16.162.215:51728] "TTL" "b5f1517db060bd92dd

program2再启动一个nextcloud

挂载数据盘
[root@program2 ~]# sudo mount -t nfs 172.16.162.217:/nfs /mnt/

[root@program2 ~]# docker run -d \
    --name nextcloud21 \
    -e MYSQL_PASSWORD=Supermao666 \
    -e MYSQL_DATABASE=nextcloud \
    -v /mnt/nextcloud1:/var/www/html \
    -v /mnt/nextdata:/var/www/html/data \
    -p 80:80 \
    --network bridge \
    nextcloud:latest
b93e787b84b47394e88398f64325540e2c54370e5a4b4f0d20aeb343f8b0ab84
[root@program2 ~]# docker ps
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS                                       NAMES
b93e787b84b4   nextcloud:latest   "/entrypoint.sh apac…"   3 seconds ago    Up 2 seconds    0.0.0.0:80->80/tcp, :::80->80/tcp           nextcloud21
545f756f22f2   redis              "docker-entrypoint.s…"   29 minutes ago   Up 29 minutes   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   redis-container
[root@program2 ~]# 

高可用测试

down掉program1的 nextcloud
nextcloud仍然可用，在高负载的情况下program2也会被轮询分担流量压力
[root@program1 opt]# docker ps
CONTAINER ID   IMAGE              COMMAND                  CREATED         STATUS         PORTS                                                  NAMES
a626845e9e90   nextcloud:latest   "/entrypoint.sh apac…"   7 seconds ago   Up 6 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp                      nextcloud21
84fe8c4fc4bf   mysql:5.7          "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mariadb
[root@program1 opt]# docker stop a62

尝试任意down掉一个节点的服务，服务仍然可用

最后只留均衡负载的弹性ip

架构仍可优化

遇到更大的数据量，可以接着横向拓展。云厂商的弹性伸缩组也是一个很好的选择

nfs在大量的写入中会变成性能瓶颈。若遇大流量可以考虑ceph或者s3

nginx-bal在遇到超大流量下，会直接down。考虑做灾备与分流