Flume性能测试报告(翻译Flume官方wiki报告)

因使用flume的时候总是会对其性能有所调研，网上找的要么就是自测的
这里找到一份官方wiki的测试报告供大家参考

https://cwiki.apache.org/confluence/display/FLUME/Performance+Measurements+-+round+2

测试环境:

以下测试基于单个agent

hadoop集群配置：20-node Hadoop cluster (1 name node and 19 data nodes).

服务器配置: 24 cores – Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM, 7200 rpm Hard Drive.

1. File channel with HDFS Sink (Sequence File):

基于1.4版本的flume测试，source为4个exec，channel为file，sink为hdfs

Flume version: 1.4

Source: 4 x Exec Source, 100k batchSize

HDFS Sink Batch size: 500,000

Event Size: 500 byte events.

Channel: File

Events/Sec
Sinks	1 data dirs	2 data dirs	4 data dirs	6 data dirs	8 data dirs	10 data dirs
1	14.3k(7Mb/s)
2	21.9k
4		35.8k
8			72.5k	77k	78.6(37Mb/s)	76.6k
10			58k
12			49.3k	49k

Measurements were taken to get an idea around the configuration that yields best performance. So took measurements only for all data points in the grid that made sense. For example it was not necessary to take measurements for multiple dataDirs at single sink, as it was evident multiple HDFS sink would better than single sink config.

混合的多sinks要比单sink的效果好

2. HDFS Sink:

相比1使用了内存channel ，memory channel

Flume version: 1.4

Channel: Memory

Event Size: 500 byte events.

#hdfs sinks	snappy batch sz:1.2mill	snappy batch sz:1.4mill	Sequence File batch sz:1.2mill
1	34.3k(17Mb/s)	33k	33k
2	71k	75k	69k
4	141k	145k	141k
8	271k	273k	251k
12	382k	380k	370k
16	478k	538k(240M/s)	486k(232M/s)

Some simple observations:

increasing number of dataDirs helps FC perf even on single disk systems
Increasing number of sinks helps

提高sink的数量是有显著效果的

3. Hive Sink:

hive sink ，channel为内存，flume版本为1.5或者1.6

Flume version: 1.5 & 1.6

Channel: Memory

BatchSz:1million

Event Size: 500 byte events.

	Flume 1.5		Flume 1.6
	Events/s	Mps	Events/s	Mps
	1 Sink
DELIMITED Text	36,885	18	138,461	66
Json	12,735	6


	16 sinks(agent maxed out)
DELIMITED Text	209,600	100	348,214	166
Json	25,751	12	31,135	14

Observation: Feeding JSON data to Hive sink is much slower, potentially due to higher parsing overhead of JSON in part.

发送json数据格式会慢一些，主要是慢在json的解析上

4. HBase Sink:

Flume version: 1.5

Channel: Memory

Serializer: RegexHbaseEventSerializer

Total Sinks: 1

Event Size(bytes)	Batch Sz:1	Batch Sz:100	Batch Sz:1000	Batch Sz:10000
500		11mb/s		11mb/s
1000	0.5bB/s	14/mb/s	22mb/s	27mb/s

5. ASync HBase Sink:

Flume version: 1.5

Channel: Memory

Serializer: SimpleAsyncHbaseEventSerializer

Total Sinks: 1

Event Size(bytes)	Batch Sz:1	Batch Sz:100	Batch Sz:1000
500		0.4mb/s	0.5mb/s
1000	0.8mb/s	0.8mb/s	0.9mb/s

6. Kafka Source:

Flume version: 1.6

Channel: Memory

Sink: Null Sink

Event Size: 1000 bytes

Total Sinks: 1

Batch Size (bytes)	Mb/s
1,000	62
10,000	112
20,000	125
40,000	147
80,000	153

作者：小闪电

出处：http://www.cnblogs.com/yueyanyu/

本文版权归作者和博客园共有，欢迎转载、交流，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文链接。如果觉得本文对您有益，欢迎点赞、欢迎探讨。本博客来源于互联网的资源，若侵犯到您的权利，请联系博主予以删除。