[转帖]MinIO系列7 - Minio性能压测
https://www.zhihu.com/people/keen-wang
前言
声明：此文为本人历史笔记的整理，文章实际撰写时间为2021年2月份，所以本中所使用的相关组件版本较老。此文是通过压力测试以理解MinIO在当前硬件条件下能提供的性能，以及相关的资源消耗情况。转载请注明出处，谢谢!
前置条件：已有Kubernetes集群，并安装好了MinIO集群。
相关组件版本：
| 组件名 | 版本 | 说明 |
|---|---|---|
| Kubernetes | 1.19 | 运行环境 |
| minio | RELEASE.2020-06-14T18-32-17Z | |
| iozone | 3-490 | 测试磁盘的吞吐量 |
| s3-benchmark | 测试MinIO的上传下载性能 |
Part 1, 测试 8Nodes/32Disks
1 集群介绍
- 8个节点，每节点4*1TiB的磁盘。磁盘底层为VMware的全SSD vSAN分布式存储。
- minio集群的节点参数设置如下
# Number of drives attached to a node
drivesPerNode: 4
# Number of MinIO containers running
replicas: 8
# Number of expanded MinIO clusters
zones: 1- pod的资源没做限制，内存request为4G
- Storage class为默认
mc admin config get minio-test storage_class
storage_class standard= rrs=EC:2 dma=write2 MinIO磁盘性能评估
2.1 测试单盘性能
- 参照官方的性能测试报告，对于单盘的性能使用dd测试
- 在k8s的node节点上找到挂载到minio的磁盘，如下面命令所示，minio使用的磁盘为/dev/sd{c.d.e.f}
kubectl get po -n minio -o wide | grep $(hostname)
minio-4 1/1 Running 0 31m 10.233.102.21 prod-ops-hy-k8s-27-1-31 <none> <none>
kubectl exec -it minio-4 -n minio -- df -Th|grep export
/dev/sde ext4 1006.9G 214.4M 955.5G 0% /export-2
/dev/sdd ext4 1006.9G 214.4M 955.5G 0% /export-3
/dev/sdc ext4 1006.9G 2.1G 953.6G 0% /export-0
/dev/sdf ext4 1006.9G 79.3M 955.6G 0% /export-1- 查看本地/dev/sdc的挂载路径
df -Th|grep sdc
/dev/sdc ext4 1007G 2.1G 954G 1% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7021959-4a18-42e6-b4c6-10eab427e5cc/globalmount- 进入上面查找到的路径，执行dd
## 以16M块连续写入2GB数据
# dd if=/dev/zero of=./test bs=16M count=128 oflag=direct
128+0 records in
128+0 records out
2147483648 bytes (2.1 GB) copied, 2.12772 s, 1.0 GB/s
## 以16M块连续读取2GB数据
# dd of=/dev/null if=./test bs=16M count=128 iflag=direct
128+0 records in
128+0 records out
2147483648 bytes (2.1 GB) copied, 2.01436 s, 1.1 GB/s- 上面结果为，单磁盘以16M块连续写吞吐量为1GB/s；以16M块连续读吞吐量为1.1GB/s。
2.2 测试JBOD性能
- 下载iozone，并安装
wget http://www.iozone.org/src/current/iozone-3-490.x86_64.rpm
rpm -ivh iozone-3-490.x86_64.rpm
ls -l /opt/iozone/bin/iozone
-rwxr-xr-x 1 root root 336464 Jul 14 2020 /opt/iozone/bin/iozone- 在/mnt下面创建挂载目录，并将/dev/sd{c.d.e.f}重新挂载，以方便测试
mkdir /mnt/drive{1..4}
mount /dev/sdc /mnt/drive1
mount /dev/sdd /mnt/drive2
mount /dev/sde /mnt/drive3
mount /dev/sdf /mnt/drive4
df -Th | grep /mnt/drive
/dev/sdd ext4 1007G 215M 956G 1% /mnt/drive2
/dev/sdc ext4 1007G 2.1G 954G 1% /mnt/drive1
/dev/sde ext4 1007G 215M 956G 1% /mnt/drive3
/dev/sdf ext4 1007G 80M 956G 1% /mnt/drive4- 按照官方性能测试报告的推荐，执行iozone，同样使用32 thread。由于不清楚官方文档中取的读写吞吐量值是Children还是Parent，当前测试出来的取Children的话，4个盘聚合的写吞吐量可达到1.2GB/s，读吞吐量为2.3GB/s。
/opt/iozone/bin/iozone -t 32 -I -r 32K -s 256M -F /mnt/drive{1..4}/tmp{1..8}
Iozone: Performance Test of File I/O
...
Children see throughput for 32 initial writers = 1290740.23 kB/sec
Parent sees throughput for 32 initial writers = 1030806.54 kB/sec
...
Children see throughput for 32 readers = 2365000.80 kB/sec
Parent sees throughput for 32 readers = 2362278.36 kB/sec- 4. 这个与dd测试的单盘1.1GB/s没太大的差距，由于上面iozone使用的是32K块，可能存在iops的性能达到了极限，而吞吐量上不去的情况。所以改成16M的块再测试一遍，结果如下所示，写吞吐量达到了2.76GB/s，读吞吐量达到了2.74GB/s。
/opt/iozone/bin/iozone -t 32 -I -r 16M -s 256M -F /mnt/drive{1..4}/tmp{1..8}
Iozone: Performance Test of File I/O
...
Children see throughput for 32 initial writers = 2895796.42 kB/sec
Parent sees throughput for 32 initial writers = 2450370.20 kB/sec
Min throughput per process = 79270.73 kB/sec
Max throughput per process = 115852.30 kB/sec
Avg throughput per process = 90493.64 kB/sec
Min xfer = 180224.00 kB
...
Children see throughput for 32 readers = 2874974.30 kB/sec
Parent sees throughput for 32 readers = 2793761.68 kB/sec
Min throughput per process = 81235.27 kB/sec
Max throughput per process = 100890.62 kB/sec
Avg throughput per process = 89842.95 kB/sec
Min xfer = 212992.00 kB3 MinIO吞吐量Benchmark
3.1 准备客户机
- 从github上克隆wasabi的s3-benchmark到ansible的主控主机上
git clone https://github.com/wasabi-tech/s3-benchmark.git
Cloning into 's3-benchmark'...
remote: Enumerating objects: 40, done.
remote: Total 40 (delta 0), reused 0 (delta 0), pack-reused 40
Unpacking objects: 100% (40/40), done.- 这们使用一个k8s测试集群的8个主机节点来做MinIO benchmark的客户机，在ansibie host中这些主机归属于ops-hy-k8s组
at /etc/ansible/hosts
[ops-hy-k8s]
prod-ops-hy-k8s-27-1-11
prod-ops-hy-k8s-27-1-12
prod-ops-hy-k8s-27-1-13
prod-ops-hy-k8s-27-1-14
prod-ops-hy-k8s-27-1-15
prod-ops-hy-k8s-27-1-16
prod-ops-hy-k8s-27-1-17
prod-ops-hy-k8s-27-1-18- 执行如下ansible命令将s3-benchmark复制到客户机
cd s3-benchmark
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark/s3-benchmark dest=/usr/local/bin/s3-benchmark mode=700"- 4. 测试下是否可以批量执行
ansible ops-hy-k8s -m shell -a "/usr/local/bin/s3-benchmark -h"
prod-ops-hy-k8s-27-1-14 | FAILED | rc=2 >>
Wasabi benchmark program v2.0Usage of myflag:
-a string
Access key
-b string
Bucket for testing (default "wasabi-benchmark-bucket")
-d int
Duration of each test in seconds (default 60)
-l int
Number of times to repeat test (default 1)
-r string
Region for testing (default "us-east-1")
-s string
Secret key
-t int
Number of threads to run (default 1)
-u string
URL for host with method prefix (default "http://s3.wasabisys.com")
-z string
Size of objects in bytes with postfix K, M, and G (default "1M")non-zero return code
...3.2 启动benchmark
- 先生成一个脚本，用来启动s3-benchmark，并复制到每个客户机。-z 2G意思是使用2G的对象进行上传下载测试，-t 32意思是客户端开启32进程
echo "/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 2G -t 32 -b s3bench-\$HOSTNAME -d 1" > /tmp/s3-benchmark.sh
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh"- 启动s3-benchmark，请注意这里一定要带"-f 8"，不然ansible只会同样启动5个客户机上的脚本
ansible ops-hy-k8s -f 8 -m shell -a "/root/s3-benchmark.sh"- 到任意客户机上查看s3-benchmark是否在运行
ps -ef|grep s3
root 4843 4815 0 22:22 pts/4 00:00:00 /bin/sh -c /root/s3-benchmark.sh
root 4844 4843 85 22:22 pts/4 00:00:02 /usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 2G -t 32 -b s3bench-prod-ops-hy-k8s-27-1-11 -d 1- 运行完成后返回结果，有两个客户机因为tcp连接重置而退出了
prod-ops-hy-k8s-27-1-13 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-13, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/08 23:12:14 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-13/Object-27: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-13/Object-27: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.13:40542->10.27.1.31:443: write: connection reset by peernon-zero return code
prod-ops-hy-k8s-27-1-17 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-17, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/08 23:12:17 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-17/Object-15: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-17/Object-15: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.17:35888->10.27.1.31:443: write: connection reset by peernon-zero return code
prod-ops-hy-k8s-27-1-18 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-18, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 231.4 secs, objects = 32, speed = 283.3MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 83.1 secs, objects = 32, speed = 789MB/sec, 0.4 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 159.8 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-12 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-12, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 366.9 secs, objects = 32, speed = 178.6MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 115.5 secs, objects = 32, speed = 567.4MB/sec, 0.3 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.2 secs, 25.9 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-11 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 429.8 secs, objects = 32, speed = 152.5MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 165.5 secs, objects = 32, speed = 396MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 137.0 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-16 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-16, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 450.1 secs, objects = 32, speed = 145.6MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 145.6 secs, objects = 32, speed = 450.1MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 160.2 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-14 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-14, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 442.7 secs, objects = 32, speed = 148MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 153.9 secs, objects = 32, speed = 425.7MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 110.2 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-15 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-15, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 441.6 secs, objects = 32, speed = 148.4MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 155.5 secs, objects = 32, speed = 421.6MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 116.1 deletes/sec. Slowdowns = 03.3. 结果分析
- 客户机吞吐量统计
| 客户机 | PUT | GET | |
|---|---|---|---|
| 节点1 | 152.5MB/s | 396MB/s | |
| 节点2 | 178.6MB/s | 567.4MB/s | |
| 节点3 | 失败 | ||
| 节点4 | 148MB/s | 425.7MB/s | |
| 节点5 | 148.4MB/s | 421.6MB/s | |
| 节点6 | 145.6MB/s | 450.1MB/s | |
| 节点7 | 失败 | ||
| 节点8 | 283.3MB/s | 789MB/ | |
| 汇总 | 1056.4MB/s | 3049.8MB/s |
- CPU、内存消耗峰值和带宽峰值
| MinIO节点 | CPU使用峰值/核 | 内存使用峰值/GiB | 带宽峰值/Rx | 带宽峰值/Tx |
|---|---|---|---|---|
| minio-0 | 1.37 | 11.13 | 354MB/s | 425MB/s |
| minio-1 | 1.31 | 11.26 | 375MB/s | 439MB/s |
| minio-2 | 1.81 | 10.73 | 375MB/s | 449MB/s |
| minio-3 | 1.12 | 10.48 | 362MB/s | 405MB/s |
| minio-4 | 1.52 | 10.91 | 402MB/s | 463MB/s |
| minio-5 | 1.16 | 10.99 | 343MB/s | 393MB/s |
| minio-6 | 2.07 | 11.42 | 362MB/s | 434MB/s |
| minio-7 | 1.57 | 10.94 | 340MB/s | 423MB/s |
| 平均 | 1.5 | 11 | 364MB/s | 429MB/s |
4 小文件benchmark
4.1 修改参数并启动benchmark
- 修改s3-benchmark的启动脚本为如下，-z 1M将对象大小修改为1M，-d 300持续测试300秒
/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 1M -t 32 -d 300 -b s3bench-$HOSTNAME- 重新复制到各个客户机
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh mode=700"- 启动脚本
ansible ops-hy-k8s -f 8 -m shell -a "sh /root/s3-benchmark.sh"4.2. 结果分析
- 客户机吞吐量统计
| 客户机 | PUT吞吐量 | PUT ops | GET吞吐量 | GET ops |
|---|---|---|---|---|
| 节点1 | 62.2MB/sec | 62.2 | 316.7MB/s | 316.7 |
| 节点2 | 62.1MB/sec | 62.1 | 320.3MB/s | 320.3 |
| 节点3 | 62.5MB/sec | 62.5 | 317.3MB/s | 317.3 |
| 节点4 | 62.4MB/sec | 62.4 | 319.2MB/s | 319.2 |
| 节点5 | 62MB/sec | 62 | 319.3MB/s | 319.3 |
| 节点6 | 62.5MB/sec | 62.5 | 300.6MB/s | 300.6 |
| 节点7 | 62.4MB/sec | 62.4 | 318.5MB/s | 318.5 |
| 节点8 | 62.6MB/sec | 62.6 | 319.5MB/s | 319.5 |
| 汇总 | 498.7MB/s | 498.7 | 2531.4MB/s | 2531.4 |
- CPU、内存消耗峰值和带宽峰值
| MinIO节点 | CPU使用峰值/核 | 内存使用峰值/GiB | 带宽峰值/Rx | 带宽峰值/Tx |
|---|---|---|---|---|
| minio-0 | 8.05 | 22.92 | 202.7MB/s | 274.9MB/s |
| minio-1 | 7.81 | 22.15 | 199.6MB/s | 276.5MB/s |
| minio-2 | 8.92 | 22.22 | 212.6MB/s | 292.4MB/s |
| minio-3 | 8.15 | 22.19 | 195MB/s | 275.1MB/s |
| minio-4 | 8.5 | 22.50 | 202.9MB/s | 280.8MB/s |
| minio-5 | 7.32 | 22.01 | 196.4MB/s | 273.2MB/s |
| minio-6 | 9.36 | 22.03 | 204.2MB/s | 278.4MB/s |
| minio-7 | 8.64 | 22.49 | 198MB/s | 268.9MB/s |
| 平均 | 8.34 | 22.31 | 201.4MB/s | 277.5MB/s |
5 测试结论
- 整个集群的大对象（2G）吞吐量，写入为1056.4MB/s，读取为3049.8MB/s。
- 小对象（1M）的写入吞吐量为498.7MB/s，写入ops为498.7；读取吞吐量为2531.4 MB/s，读取ops为2531.4。
- 对于单节点4个磁盘的JBOD吞吐量，读写能达到2.7GB/s，8个节点的集群理论IO吞吐量能达到21.6GB/s，节点间的网络吞吐量为25Gb/s的硬件环境。这个性能只能说差强人意，也可能是由于没有针对性优化。
Part 2，测试有资源限制的8Nodes/32Disks
1 集群介绍
- 在《Part1， 测试8Nodes/32Disks》中我们可以看到并发上升后，minio的资源使用还是比较厉害。在小文件压测中，集群中节点的CPU平均使用8.34核，内存平均使用为22.31GiB。正式生产使用中，如果我们不需要到这么高的吞吐量，但是资源不做限制时，pod很容易就使用过多的资源。
- 因此，本次测试我们将资源限制为如下，以观察这些资源能支撑多大的并发量
resources:
requests:
cpu: 1
memory: 2Gi
limits:
cpu: 2
memory: 4Gi- 其他保持一致，然后重新构建集群
2 MinIO磁盘性能评估
磁盘性能与《测试1-8Nodes-32Disks》中的保持一致
3 吞吐量Benchmark
3.1 准备客户机
请参照《Part1， 测试8Nodes/32Disks》中的3.1
3.2 启动benchmark
- 运行完成后返回结果，有8个客户机中有7个为tcp连接重置而退出了，只有1个执行成功
prod-ops-hy-k8s-27-1-11 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/10 12:58:05 WARNING: createBucket s3bench-prod-ops-hy-k8s-27-1-11 error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
status code: 409, request id: 16624A187E2ED504, host id:
2021/02/10 12:59:35 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-8: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-8: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.11:51916->10.27.1.31:443: write: connection reset by peernon-zero return code
...
prod-ops-hy-k8s-27-1-16 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-16, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 197.5 secs, objects = 32, speed = 331.9MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: GET time 70.4 secs, objects = 32, speed = 931.3MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.4 secs, 77.0 deletes/sec. Slowdowns = 02021/02/10 12:58:06 WARNING: createBucket s3bench-prod-ops-hy-k8s-27-1-16 error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
status code: 409, request id: 16624A1896EE10BC, host id:- 查看ingress nginx的日志
2021/02/10 04:59:18 [error] 46#46: *29201 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.17, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-17/Object-11 HTTP/1.1", upstream: "http://10.233.119.28:9000/s3bench-prod-ops-hy-k8s-27-1-17/Object-11", host: "minio-test.xxx.com"
2021/02/10 04:59:20 [error] 43#43: *29225 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.17, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-17/Object-23 HTTP/1.1", upstream: "http://10.233.107.26:9000/s3bench-prod-ops-hy-k8s-27-1-17/Object-23", host: "minio-test.xxx.com"
I0210 04:59:21.486939 7 main.go:187] "Received SIGTERM, shutting down"
I0210 04:59:21.486977 7 nginx.go:372] "Shutting down controller queues"
I0210 04:59:21.740124 7 nginx.go:380] "Stopping admission controller"
I0210 04:59:21.740262 7 nginx.go:388] "Stopping NGINX process"
E0210 04:59:21.740321 7 nginx.go:319] "Error listening for TLS connections" err="http: Server closed"
2021/02/10 04:59:21 [notice] 590#590: signal process started
2021/02/10 04:59:26 [error] 87#87: *29196 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.17, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-17/Object-18 HTTP/1.1", upstream: "http://10.233.112.30:9000/s3bench-prod-ops-hy-k8s-27-1-17/Object-18", host: "minio-test.xxx.com"
2021/02/10 04:59:28 [error] 130#130: *29130 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.14, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-14/Object-12 HTTP/1.1", upstream: "http://10.233.114.25:9000/s3bench-prod-ops-hy-k8s-27-1-14/Object-12", host: "minio-test.xxx.com"- 原因是ingress nginx的proxy_connect_timeout默认是5s，5s连不上upstream host就会超时，通过kubectl edit ingress的方式修改名为minio的ingress，添加proxy-connect-timeout、proxy-read-timeout和proxy-send-timeout这3个annotation
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
...
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "120"
nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"- 然后重启测试，仍有4个客户机失败，4个成功
prod-ops-hy-k8s-27-1-11 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/10 13:17:36 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-20: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-20: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.11:36496->10.27.1.34:443: write: connection reset by peernon-zero return code
prod-ops-hy-k8s-27-1-17 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-17, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 333.7 secs, objects = 32, speed = 196.4MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 70.3 secs, objects = 32, speed = 932.3MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.6 secs, 51.0 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-13 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-13, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 342.6 secs, objects = 32, speed = 191.3MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 70.0 secs, objects = 32, speed = 936.7MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.0 secs, 33.4 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-18 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-18, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 375.0 secs, objects = 32, speed = 174.8MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 74.9 secs, objects = 32, speed = 875.1MB/sec, 0.4 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 94.5 deletes/sec. Slowdowns = 0
prod-ops-hy-k8s-27-1-14 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-14, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 413.1 secs, objects = 32, speed = 158.7MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 69.1 secs, objects = 32, speed = 947.9MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.4 secs, 74.8 deletes/sec. Slowdowns = 0- 修改ansible的host配置文件，增加一个主机组为s3-client，将客户机的数量减少至4个后所有都执行成功了
[s3-client]
prod-ops-hy-k8s-27-1-11
prod-ops-hy-k8s-27-1-12
prod-ops-hy-k8s-27-1-13
prod-ops-hy-k8s-27-1-143.3. 结果分析
- 客户机吞吐量统计
| 客户机 | PUT | GET | |
|---|---|---|---|
| 节点1 | 254.5MB/sec | 669.3MB/sec | |
| 节点2 | 443.7MB/sec | 755.1MB/sec | |
| 节点3 | 238.7MB/sec | 717.6MB/sec | |
| 节点4 | 231.9MB/sec | 779.7MB/sec | |
| 汇总 | 1168.8MB/s | 2921.7MB/s |
- CPU、内存消耗峰值和带宽峰值
| MinIO节点 | CPU使用峰值/核 | 内存使用峰值/GiB | 带宽峰值/Rx | 带宽峰值/Tx |
|---|---|---|---|---|
| minio-0 | 1.16 | 3.17 | 299MB/s | 354MB/s |
| minio-1 | 1.65 | 3.25 | 353MB/s | 414MB/s |
| minio-2 | 1.71 | 3.24 | 268MB/s | 331MB/s |
| minio-3 | 1.57 | 3.35 | 314MB/s | 352MB/s |
| minio-4 | 1.46 | 3.1 | 293MB/s | 308MB/s |
| minio-5 | 1.34 | 3.07 | 208MB/s | 337MB/s |
| minio-6 | 1.5 | 3.17 | 288MB/s | 316MB/s |
| minio-7 | 1.81 | 3.11 | 327MB/s | 392MB/s |
| 平均 | 1.53 | 3.18 | 294MB/s | 350MB/s |
4 小文件benchmark
4.1 修改参数并启动benchmark
- 修改s3-benchmark的启动脚本为如下，-z 1M将对象大小修改为1M/100K，-d 300持续测试300秒
/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 1M -t 32 -d 300 -b s3bench-$HOSTNAME- 重新复制到各个客户机
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh mode=700"- 启动脚本
ansible ops-hy-k8s -f 8 -m shell -a "sh /root/s3-benchmark.sh"4.2 结果分析
4.2.1 1M对象大小
- 客户机吞吐量统计
| 客户机 | PUT吞吐量 | PUT ops | GET吞吐量 | GET ops |
|---|---|---|---|---|
| 节点1 | 19.5MB/sec | 19.5 | 50.8MB/sec | 50.8 |
| 节点2 | 19.7MB/sec | 19.7 | 50.2MB/sec | 50.2 |
| 节点3 | 19.5MB/sec | 19.5 | 50.5MB/sec | 50.5 |
| 节点4 | 19.4MB/sec | 19.4 | 50.4MB/sec | 50.4 |
| 节点5 | 19.5MB/sec | 19.5 | 50.6MB/sec | 50.6 |
| 节点6 | 19.6MB/sec | 19.6 | 50.2MB/sec | 50.2 |
| 节点7 | 19.6MB/sec | 19.6 | 51MB/sec | 51 |
| 节点8 | 19.5MB/sec | 19.5 | 50.5MB/sec | 50.5 |
| 汇总 | 156.3MB/s | 156.3 | 404.2MB/s | 404.2 |
- CPU、内存消耗峰值和带宽峰值
| MinIO节点 | CPU使用峰值/核 | 内存使用峰值/GiB | 带宽峰值/Rx | 带宽峰值/Tx |
|---|---|---|---|---|
| minio-0 | 1.81 | 2.82 | 93.5MB/s | 102.4MB/s |
| minio-1 | 1.71 | 2.96 | 68.2MB/s | 114.2MB/s |
| minio-2 | 2 | 3.1 | 109.3MB/s | 177.4MB/s |
| minio-3 | 1.69 | 2.8 | 111.1MB/s | 140.6MB/s |
| minio-4 | 1.91 | 2.97 | 69.7MB/s | 112.6MB/s |
| minio-5 | 1.65 | 2.92 | 79.6MB/s | 102MB/s |
| minio-6 | 1.88 | 2.9 | 79.5MB/s | 102.7MB/s |
| minio-7 | 1.87 | 2.79 | 96MB/s | 197.2MB/s |
| 平均 | 1.82 | 2.91 | 87.6MB/s | 135.2MB/s |
4.2.2 100KB对象大小
- 客户机吞吐量统计
| 客户机 | PUT吞吐量 | PUT ops | GET吞吐量 | GET ops |
|---|---|---|---|---|
| 节点1 | 2.4MB/sec | 24.3 | 5.3MB/sec | 54.7 |
| 节点2 | 2.4MB/sec | 24.6 | 5.4MB/sec | 55.2 |
| 节点3 | 2.4MB/sec | 24.3 | 5.4MB/sec | 55.5 |
| 节点4 | 2.4MB/sec | 24.6 | 5.4MB/sec | 55 |
| 节点5 | 2.4MB/sec | 24.6 | 5.3MB/sec | 54.8 |
| 节点6 | 2.4MB/sec | 24.4 | 5.4MB/sec | 55.4 |
| 节点7 | 2.4MB/sec | 24.6 | 5.3MB/sec | 54.8 |
| 节点8 | 2.4MB/sec | 24.6 | 5.4MB/sec | 55.2 |
| 汇总 | 19.2MB/s | 196 | 42.9MB/s | 440.6 |
- CPU、内存消耗峰值和带宽峰值
| MinIO节点 | CPU使用峰值/核 | 内存使用峰值/GiB | 带宽峰值/Rx | 带宽峰值/Tx |
|---|---|---|---|---|
| minio-0 | 1.78 | 3.29 | 12.83MB/s | 17.53MB/s |
| minio-1 | 1.72 | 3.31 | 14.93MB/s | 20.09MB/s |
| minio-2 | 2 | 3.5 | 19.93MB/s | 25.49MB/s |
| minio-3 | 1.71 | 3.09 | 18.36MB/s | 23.36MB/s |
| minio-4 | 1.89 | 3.42 | 17.03MB/s | 15.99MB/s |
| minio-5 | 1.68 | 3.07 | 13.45MB/s | 15.63MB/s |
| minio-6 | 1.89 | 3.29 | 14.13MB/s | 16.37MB/s |
| minio-7 | 1.83 | 3.17 | 16.21MB/s | 25.99MB/s |
| Avg | 1.81 | 3.27 | 15.86MB/s | 20.06MB/s |
5 测试结论
- 大对象的吞吐量数量与无资源限制相差不多。与小对象相比，操作数少很多，会造成资源消耗相对少不少，特别是内存。
- 1M对象和100K对象的ops性能基本在一个量级，100K略高。CPU使用率基本相同，100K的内存使用率略高。可见在磁盘的IO性能未达瓶颈时，小对象的ops处决于CPU/内存资源，与对象大小相关性不大。
- 在设计MinIO生产集群时，应考虑如下几点：
- 先预先评估集群的存储对象大小，是大对象为主（>100M），还是小对象为主（KB ~ MB）。
- 当以大对象为主时，应重点关注磁盘的吞吐量；当以小对象为主时，应重点关注磁盘的IOPS和MinIO节点的内存。
- 然后定义MinIO集群的性能指标需求，再平衡的考虑磁盘性能，以及CPU/内存资源的配备。
[转帖]MinIO系列7 - Minio性能压测的更多相关文章
- jmeter系列-如何实现像loadrunner一样，多个并发用户先通过登录初始化，然后做并发的接口性能压测
自动转开发后,就很少关注性能测试方面的东西,最近在帮朋友做一个性能压测,由于朋友那边的公司比较小,环境比较简单,而且是对http服务进行的压测,所以最终 选用了jmeter来实现这个压测. 如下就是我 ...
- 性能压测中的SLA，你知道吗？
本文是<Performance Test Together>(简称PTT)系列专题分享的第6期,该专题将从性能压测的设计.实现.执行.监控.问题定位和分析.应用场景等多个纬度对性能压测的全 ...
- 并发模式与 RPS 模式之争，性能压测领域的星球大战
本文是<如何做好性能压测>系列专题分享的第四期,该专题将从性能压测的设计.实现.执行.监控.问题定位和分析.应用场景等多个纬度对性能压测的全过程进行拆解,以帮助大家构建完整的性能压测的理论 ...
- 性能压测诡异的Requests/second 响应刺尖问题
最近一段时间都在忙着转java项目最后的冲刺,前期的coding翻代码.debug.fixbug都逐渐收尾,进入上线前的性能压测. 虽然不是大促前的性能压测要求,但是为了安全起见,需要摸个底心里有个数 ...
- 性能压测，SQL查询异常
早上测试对性能压测,发现取sequence服务大量超时报错,查询线上的监控SQL: 大量这个查询,我在DeviceID和Isdelete上建有复合索引,应该很快,而且我测试了一下,取值,执行效率很高, ...
- jmeter性能压测瓶颈排查-网络带宽
问题: 有一台机器做性能压测的时候,无论开多少个线程,QPS一直压不上去,而服务器和数据库的性能指标(主要是CPU和内存)一直维持在很低的水平. 希望帮忙排查一下原因. 过去看了下进行压测的接口代码, ...
- [SCF+wetest+jmeter]简单云性能压测工具使用方案
前言 压测太难?局域网压力无法判断服务器网络指标?无法产生非常大的并发量?云性能太贵? 也许我们可以把各种简单的工具拼起来进行压力测试! 准备 https://cloud.tencent.com/pr ...
- 软件性能测试分析与调优实践之路-JMeter对RPC服务的性能压测分析与调优-手稿节选
一.JMeter 如何通过自定义Sample来压测RPC服务 RPC(Remote Procedure Call)俗称远程过程调用,是常用的一种高效的服务调用方式,也是性能压测时经常遇到的一种服务调用 ...
- MySQL 性能压测工具-sysbench，从入门到自定义测试项
sysbench是一个开源的.基于LuaJIT(LuaJIT 是 Lua 的即时编译器,可将代码直接翻译成机器码,性能比原生 lua 要高) 的.可自定义脚本的多线程基准测试工具,也是目前用得最多的 ...
- EMQ X 系统调优和性能压测
前言 如果使用 EMQ 来承载百万级别的用户连接可以吗?毕竟在 MQTT 官方介绍上说 EMQ X 可以处理千万并发客户端,而 EMQ X 自己官方称 4.x 版本 MQTT 连接压力测试一台 8 核 ...
随机推荐
- 【K8S系列】如何高效查看 k8s日志
序言 你只管努力,其他交给时间,时间会证明一切. 文章标记颜色说明: 黄色:重要标题 红色:用来标记结论 绿色:用来标记一级论点 蓝色:用来标记二级论点 Kubernetes (k8s) 是一个容器编 ...
- Java数组中常见的方法
一.前言 代码: //给定一个数组 int[] arr = {234,312,32,1321,321,43}; int[] arr1 = new int[6]; int[] arr2 = {1,3,7 ...
- php +libcurl+nghttp2 实现高性能微服务架构
1.server端nginx编译时增加参数configure --with-http_v2_module server { listen 80 http2; ...
- MyBatis中SQL语句优化小结
摘要:MyBatis 作为一款优秀的持久层框架,它支持自定义SQL.存储过程以及高级映射. MyBatis 作为一款优秀的持久层框架,它支持自定义SQL.存储过程以及高级映射.它免除了几乎所有的 JD ...
- PNG文件解读(2):PNG格式文件结构与数据结构解读—解码PNG数据
PNG文件识别 之前写过<JPEG/Exif/TIFF格式解读(1):JEPG图片压缩与存储原理分析>,JPEG文件是以,FFD8开头,FFD9结尾,中间存储着以0xFFE0~0xFFEF ...
- 剖析字节案例，火山引擎 A/B 测试 DataTester 如何“嵌入”技术研发流程
更多技术交流.求职机会,欢迎关注字节跳动数据平台微信公众号,回复[1]进入官方交流群 日前,在 WOT 全球创新技术大会上,火山引擎 DataTester 技术负责人韩云飞做了关于字节跳动 A/B 测 ...
- Kubernetes(K8S) 介绍
Master Api Server 统一入口,以 Restful 方式,交给 etcd 存储 Scheduler 节点调试,选择 Node 节点,做应用部署 Controller Manager 处理 ...
- 将镜像上传到Docker Hub中央仓库中
首先创建一个镜像,点击:创建一个简单的Docker镜像 1.先注册帐号 https://hub.docker.com/ 2.将镜象推上去 [root@localhost docker]# docker ...
- ajax补充说明 多对多三种创建方式 django内置序列化组件 ORM批量操作数据 分页器 form组件入门
目录 ajax补充说明 request.is_ajax() ajax回调函数接收返回值 ajax回调函数 接受json数据 第一种方式:后端使用json模块 第二种方式:后端返回JsonRespons ...
- Java 使用 slf4j + log4j 写日志
没有SpringBoot等框架的情况下 pom.xml: <properties> <slf4j.version>1.7.26</slf4j.version> &l ...