kafka学习之五_多个磁盘的性能验证

kafka学习之五_多个磁盘的性能验证

---

背景

周末在家学习kafka
上午验证了grafana+kafka_exporter的监控
下午想着验证一把性能相关. kafka学习之三里面,有成套的脚本.
我这边想起来之前还有一个机器, 是四个单盘HDD, 我可以直接进行使用和验证.

---

测试思路

计划创建四个partition的topic进行测试
首先仅使用第一个磁盘进行验证
测试完删除掉, 重新初始化, log,dirs 增加多个分区的的挂载路径.
然后进行相关的测试验证

---

测试结果分析

自己尝试进行了一些验证.
发现1块磁盘和四块磁盘, 如果使用一个producer-perf 进行测试时, 测试结果差异性不是很大.
可能跟CPU和磁盘的性能压力都有关系. 

如果开启多个topic, 同时使用多个 producer-perf进行测试时. 总体的吞吐量, 四块硬盘比一块硬盘大约高50%
压测的通知开启 iostat 验证
单盘时压力都在 sdc 上面
四块盘时 压力其实不是非常平均, 这也是造成压测结果不是线性提升的很大原因.
单盘时四个topic 同时压测.总计为: 96.99
四盘时四个topic 同时压测.总计为: 140.78

总体的性能比率为: 45%

---

磁盘信息

/dev/sdc                ext4      1.8T  539G  1.2T   31% /hdd1
/dev/sdf                xfs       1.9T  4.6G  1.9T    1% /hdd4
/dev/sde                xfs       1.9T  1.9G  1.9T    1% /hdd3
/dev/sdd                xfs       1.9T  380G  1.5T   21% /hdd2

---

配置文件的差异

第一次测试时:
log.dirs=/hdd1/kafkahdd
第二次测试时
log.dirs=/hdd1/kafkahdd2,/hdd2/kafkahdd2,/hdd3/kafkahdd2,/hdd4/kafkahdd2
注意: 如果是四个目录时 会这样:
Formatting /hdd1/kafkahdd2 with metadata.version 3.5-IV2.
Formatting /hdd2/kafkahdd2 with metadata.version 3.5-IV2.
Formatting /hdd3/kafkahdd2 with metadata.version 3.5-IV2.
Formatting /hdd4/kafkahdd2 with metadata.version 3.5-IV2.

---

初始化和创建topic

cd /root/kafka_2.13-3.5.0
bin/kafka-server-stop.sh  config/kraft/server.properties
bin/kafka-storage.sh format -t 7ONT3dn3RWWNCZyIwLrEqg  -c config/kraft/server.properties
bin/kafka-server-start.sh -daemon  config/kraft/server.properties
bin/kafka-topics.sh --create  --command-config config/sasl.conf  --replication-factor 1 --partitions 4 --topic zhaobsh01 --bootstrap-server 127.0.0.1:9093

---

测试命令

# 生产者测试
bin/kafka-producer-perf-test.sh  --num-records 200000  --record-size 1024 --throughput -1 --producer.config config/sasl.conf  --topic zhaobsh01  --print-metrics --producer-props bootstrap.servers=127.0.0.1:9093
# 单盘时的测试结果
200000 records sent, 36812.074360 records/sec (35.95 MB/sec), 638.59 ms avg latency, 950.00 ms max latency, 680 ms 50th, 899 ms 95th, 937 ms 99th, 946 ms 99.9th.
# 四盘时的测试结果
200000 records sent, 36172.906493 records/sec (35.33 MB/sec), 646.08 ms avg latency, 971.00 ms max latency, 705 ms 50th, 909 ms 95th, 952 ms 99th, 968 ms 99.9th.

# 消费者测试
bin/kafka-consumer-perf-test.sh  --fetch-size 10000 --messages 2000000   --topic zhaobsh01 --consumer.config config/sasl.conf   --print-metrics --bootstrap-server 127.0.0.1:9093
# 单盘时的测试结果
2023-06-24 14:03:49:723, 2023-06-24 14:04:13:453, 195.3125, 8.2306, 200000, 8428.1500, 4282, 19448, 10.0428, 10283.8338
# 四盘时的测试结果
2023-06-24 14:14:20:733, 2023-06-24 14:14:44:324, 195.3125, 8.2791, 200000, 8477.8093, 4889, 18702, 10.4434, 10694.0434

# 增加四盘时 增加io_threads 到16, network到6
# 生产者测试
200000 records sent, 37622.272385 records/sec (36.74 MB/sec), 589.76 ms avg latency, 889.00 ms max latency, 587 ms 50th, 843 ms 95th, 876 ms 99th, 885 ms 99.9th.
# 消费者测试
2023-06-24 14:25:03:258, 2023-06-24 14:25:26:604, 195.3125, 8.3660, 200000, 8566.7780, 3779, 19567, 9.9817, 10221.2909

---

四个topic同时测试时的情况

# 单一磁盘时
grep -ir "200000 records sent"
1.txt:200000 records sent, 24573.043371 records/sec (24.00 MB/sec), 977.14 ms avg latency, 1330.00 ms max latency, 995 ms 50th, 1263 ms 95th, 1313 ms 99th, 1324 ms 99.9th.
2.txt:200000 records sent, 24919.013207 records/sec (24.33 MB/sec), 966.64 ms avg latency, 1309.00 ms max latency, 1001 ms 50th, 1247 ms 95th, 1289 ms 99th, 1304 ms 99.9th.
3.txt:200000 records sent, 24479.804162 records/sec (23.91 MB/sec), 940.11 ms avg latency, 1281.00 ms max latency, 992 ms 50th, 1244 ms 95th, 1268 ms 99th, 1276 ms 99.9th.
4.txt:200000 records sent, 25348.542459 records/sec (24.75 MB/sec), 847.37 ms avg latency, 1425.00 ms max latency, 821 ms 50th, 1274 ms 95th, 1328 ms 99th, 1335 ms 99.9th.

# 四块磁盘时
grep -ir "200000 records sent"
1.txt:200000 records sent, 38044.512079 records/sec (37.15 MB/sec), 545.91 ms avg latency, 895.00 ms max latency, 572 ms 50th, 653 ms 95th, 663 ms 99th, 667 ms 99.9th.
2.txt:200000 records sent, 34668.053389 records/sec (33.86 MB/sec), 621.67 ms avg latency, 868.00 ms max latency, 658 ms 50th, 810 ms 95th, 847 ms 99th, 851 ms 99.9th.
3.txt:200000 records sent, 37383.177570 records/sec (36.51 MB/sec), 584.06 ms avg latency, 828.00 ms max latency, 594 ms 50th, 747 ms 95th, 761 ms 99th, 767 ms 99.9th.
4.txt:200000 records sent, 34059.945504 records/sec (33.26 MB/sec), 562.29 ms avg latency, 1297.00 ms max latency, 566 ms 50th, 796 ms 95th, 808 ms 99th, 812 ms 99.9th.

---

四个topic同时测试时的情况-第二次测试(提升有限)

# 单一磁盘时
grep -ir "800000 records sent"
1.txt:800000 records sent, 42413.317782 records/sec (41.42 MB/sec), 617.73 ms avg latency, 1633.00 ms max latency, 574 ms 50th, 926 ms 95th, 1613 ms 99th, 1630 ms 99.9th.
2.txt:800000 records sent, 42753.313382 records/sec (41.75 MB/sec), 619.57 ms avg latency, 1315.00 ms max latency, 586 ms 50th, 939 ms 95th, 1301 ms 99th, 1310 ms 99.9th.
3.txt:800000 records sent, 45300.113250 records/sec (44.24 MB/sec), 571.20 ms avg latency, 1321.00 ms max latency, 564 ms 50th, 841 ms 95th, 1299 ms 99th, 1315 ms 99.9th.
4.txt:800000 records sent, 43773.254541 records/sec (42.75 MB/sec), 596.18 ms avg latency, 1329.00 ms max latency, 570 ms 50th, 931 ms 95th, 1320 ms 99th, 1326 ms 99.9th.

# 四块磁盘时
grep -ir "800000 records sent"
1.txt:800000 records sent, 47292.504138 records/sec (46.18 MB/sec), 571.88 ms avg latency, 924.00 ms max latency, 580 ms 50th, 814 ms 95th, 900 ms 99th, 918 ms 99.9th.
2.txt:800000 records sent, 46794.571830 records/sec (45.70 MB/sec), 551.35 ms avg latency, 1138.00 ms max latency, 510 ms 50th, 925 ms 95th, 1102 ms 99th, 1130 ms 99.9th.
3.txt:800000 records sent, 47092.064987 records/sec (45.99 MB/sec), 568.98 ms avg latency, 1095.00 ms max latency, 534 ms 50th, 862 ms 95th, 998 ms 99th, 1083 ms 99.9th.
4.txt:800000 records sent, 46114.825917 records/sec (45.03 MB/sec), 459.42 ms avg latency, 1374.00 ms max latency, 452 ms 50th, 959 ms 95th, 992 ms 99th, 1009 ms 99.9th.

---

所有测试脚本

bin/kafka-topics.sh --create  --command-config config/sasl.conf  --replication-factor 1 --partitions 4 --topic zhaobsh02 --bootstrap-server 127.0.0.1:9093
bin/kafka-topics.sh --create  --command-config config/sasl.conf  --replication-factor 1 --partitions 4 --topic zhaobsh03 --bootstrap-server 127.0.0.1:9093
bin/kafka-topics.sh --create  --command-config config/sasl.conf  --replication-factor 1 --partitions 4 --topic zhaobsh04 --bootstrap-server 127.0.0.1:9093
bin/kafka-topics.sh --create  --command-config config/sasl.conf  --replication-factor 1 --partitions 4 --topic zhaobsh05 --bootstrap-server 127.0.0.1:9093

nohup bin/kafka-producer-perf-test.sh  --num-records 800000  --record-size 1024 --throughput -1 --producer.config config/sasl.conf  --topic zhaobsh02  --print-metrics --producer-props bootstrap.servers=127.0.0.1:9093 >log/1.txt &
nohup bin/kafka-producer-perf-test.sh  --num-records 800000  --record-size 1024 --throughput -1 --producer.config config/sasl.conf  --topic zhaobsh03  --print-metrics --producer-props bootstrap.servers=127.0.0.1:9093 >log/2.txt &
nohup bin/kafka-producer-perf-test.sh  --num-records 800000  --record-size 1024 --throughput -1 --producer.config config/sasl.conf  --topic zhaobsh04  --print-metrics --producer-props bootstrap.servers=127.0.0.1:9093 >log/3.txt &
nohup bin/kafka-producer-perf-test.sh  --num-records 800000  --record-size 1024 --throughput -1 --producer.config config/sasl.conf  --topic zhaobsh05  --print-metrics --producer-props bootstrap.servers=127.0.0.1:9093 >log/4.txt &

grep -ir "800000 records sent"