#创建test topic 3分区1副本, test63 6分区 3副本。(因为kafka-producer-perf-test.sh未找到控制控制消息副本同步的参数,这里多副本的topic不进行测试)
[root@k8s-n2 bin]# ./kafka-topics.sh --create --zookeeper 192.168.0.126:2181,192.168.0.127:2181,192.168.0.128:2181 --topic test --partitions 3 --replication-factor 1
Created topic test.
[root@k8s-n2 kafka]# bin/kafka-topics.sh --zookeeper 192.168.0.126:2181,192.168.0.127:2181,192.168.0.128:2181 --create --topic test63 --partitions 6 --replication-factor 3
Created topic test63.
[root@k8s-n2 kafka]# bin/kafka-producer-perf-test.sh --topic test --producer-props bootstrap.servers=192.168.0.126:9092,192.168.0.127:9092,192.168.0.128:9092 --num-records 5000000 --throughput 1000000 --record-size 1000
768737 records sent, 153747.4 records/sec (146.62 MB/sec), 189.9 ms avg latency, 394.0 ms max latency.
966773 records sent, 193354.6 records/sec (184.40 MB/sec), 167.9 ms avg latency, 332.0 ms max latency.
997272 records sent, 199454.4 records/sec (190.21 MB/sec), 164.8 ms avg latency, 319.0 ms max latency.
967146 records sent, 193429.2 records/sec (184.47 MB/sec), 168.8 ms avg latency, 306.0 ms max latency.
930066 records sent, 186013.2 records/sec (177.40 MB/sec), 176.6 ms avg latency, 318.0 ms max latency.
5000000 records sent, 184945.441095 records/sec (176.38 MB/sec), 173.22 ms avg latency, 394.00 ms max latency, 212 ms 50th, 302 ms 95th, 327 ms 99th, 358 ms 99.9th.
[root@k8s-n2 kafka]# bin/kafka-producer-perf-test.sh --topic test --producer-props bootstrap.servers=192.168.0.126:9092,192.168.0.127:9092,192.168.0.128:9092 --num-records 5000000 --throughput 1000000 --record-size 500
1256801 records sent, 251360.2 records/sec (119.86 MB/sec), 227.9 ms avg latency, 377.0 ms max latency.
1856768 records sent, 371353.6 records/sec (177.08 MB/sec), 176.3 ms avg latency, 294.0 ms max latency.
1859712 records sent, 371942.4 records/sec (177.36 MB/sec), 175.5 ms avg latency, 351.0 ms max latency.
5000000 records sent, 330054.789095 records/sec (157.38 MB/sec), 189.46 ms avg latency, 377.00 ms max latency, 200 ms 50th, 311 ms 95th, 341 ms 99th, 362 ms 99.9th.
[root@k8s-n2 kafka]# bin/kafka-producer-perf-test.sh --topic test --producer-props bootstrap.servers=192.168.0.126:9092,192.168.0.127:9092,192.168.0.128:9092 --num-records 5000000 --throughput 1000000 --record-size 100
4575154 records sent, 915030.8 records/sec (87.26 MB/sec), 238.4 ms avg latency, 429.0 ms max latency.
5000000 records sent, 935278.713056 records/sec (89.20 MB/sec), 233.40 ms avg latency, 429.00 ms max latency, 236 ms 50th, 382 ms 95th, 408 ms 99th, 427 ms 99.9th.
说明:--topic指定队列名称,--producer-props bootstrap.servers=指定集群,--num-records指定要发送的消息数量,--record-size每条消息的大小,--throughput每秒最大发送的最大消息数量(-1为不限制每秒最大数量)
[root@k8s-n2 kafka]# bin/kafka-producer-perf-test.sh --topic test --producer-props bootstrap.servers=192.168.0.126:9092,192.168.0.127:9092,192.168.0.128:9092 --num-records 10000000 --throughput -1 --record-size 100
1088276 records sent, 217481.2 records/sec (20.74 MB/sec), 866.5 ms avg latency, 1786.0 ms max latency.
1958182 records sent, 390153.8 records/sec (37.21 MB/sec), 791.1 ms avg latency, 1523.0 ms max latency.
2673824 records sent, 534764.8 records/sec (51.00 MB/sec), 569.8 ms avg latency, 1085.0 ms max latency.
2984834 records sent, 596847.4 records/sec (56.92 MB/sec), 507.8 ms avg latency, 977.0 ms max latency.
10000000 records sent, 453638.178189 records/sec (43.26 MB/sec), 618.73 ms avg latency, 1786.00 ms max latency, 647 ms 50th, 1321 ms 95th, 1546 ms 99th, 1732 ms 99.9th.
每台机器每秒平均处理45条消息,3台虚拟机总共每秒生产135w条数据。通过kafka生产消息很容易每秒钟发送数量上200w的。
#使用新的topic测试,因为test中消息大小不一致,这里使用test63测试生产与消费
[root@k8s-n2 kafka]# bin/kafka-producer-perf-test.sh --topic test63 --producer-props bootstrap.servers=192.168.0.126:9092,192.168.0.127:9092,192.168.0.128:9092 --num-records 10000000 --throughput -1 --record-size 100
2879529 records sent, 575905.8 records/sec (54.92 MB/sec), 301.0 ms avg latency, 1391.0 ms max latency.
5310720 records sent, 1062144.0 records/sec (101.29 MB/sec), 287.4 ms avg latency, 1463.0 ms max latency.
10000000 records sent, 847529.451648 records/sec (80.83 MB/sec), 293.45 ms avg latency, 1463.00 ms max latency, 33 ms 50th, 1055 ms 95th, 1366 ms 99th, 1441 ms 99.9th.
[root@k8s-n2 kafka]# bin/kafka-consumer-perf-test.sh --broker-list 192.168.0.126:9092,192.168.0.127:9092,192.168.0.128:9092 --topic test63 --messages 10000000 --fetch-size 1048576 --threads 6
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-05-21 10:22:25:678, 2020-05-21 10:22:31:492, 953.6743, 164.0307, 10000000, 1719986.2401, 1590027746103, -1590027740289, -0.0000, -0.0063
start.time开始时间,end.time结束时间,data.consumed.in.MB总消费大小953.6743MB,每秒消费164.0307MB,消费总条数10000000,每秒消费1719986.2401条