flume实现kafka到hdfs测试用例

kafka 到hdfs

at1.sources =st1
at1.channels = ct1
at1.sinks = kt1

# For each one of the sources, the type is defined
at1.sources.st1.type = org.apache.flume.source.kafka.KafkaSource
at1.sources.st1.kafka.bootstrap.servers = node0.***:,node1.***:,node2.***:
at1.sources.st1.kafka.topics = flume-test
at1.sources.st1.kafka.consumer.group.id = flume-test-group
# The channel can be defined as follows.
at1.sources.st1.channels = ct1
at1.sources.st1.kafka.batchSize = 

# Each sink's type must be defined
at1.sinks.kt1.type = hdfs
at1.sinks.kt1.hdfs.path = hdfs://********/tmp/flume/test-data
at1.sinks.kt1.hdfs.fileType = DataStream

#Specify the channel the sink should use
at1.sinks.kt1.channel = ct1

# Each channel's type is defined.
at1.channels.ct1.type = memory
at1.channels.ct1.capacity =
at1.channels.ct1.transactionCapacity =
# Other config values specific to each type of channel(sink or source)
at1.sources.st1.channels = ct1
at1.sinks.kt1.channel = ct1

在该文件所存的目录下，执行以下命令调起：

flume-ng agent -name at1  -conf-file test-kafka-to-hdfs.properties