flume实现kafka到文件测试用例

kafka 到 file

at2.sources =st2
at2.channels = ct2
at2.sinks = kt2

# For each one of the sources, the type is defined
at2.sources.st2.type = org.apache.flume.source.kafka.KafkaSource
at2.sources.st2.kafka.bootstrap.servers = node0.***:9200,node1.***:9200,node2.***:9200
at2.sources.st2.kafka.topics = flume-test
at2.sources.st2.kafka.consumer.group.id = flume-to-file-test-group-id
# The channel can be defined as follows.
at2.sources.st2.channels = ct2
at2.sources.st2.kafka.batchSize = 100

# Each sink's type must be defined
at2.sinks.kt2.type = file_roll
at2.sinks.kt2.sink.directory = /mnt/home/admin/install/flume/examples/file-dir
at2.sinks.kt2.batchSize = 100
at2.sinks.kt2.sink.rollInterval = 300
#Specify the channel the sink should use
at2.sinks.kt2.channel = ct2

# Each channel's type is defined.
at2.channels.ct2.type = memory
at2.channels.ct2.capacity = 1000
at2.channels.ct2.transactionCapacity = 1000
# Other config values specific to each type of channel(sink or source)
at2.sources.st2.channels = ct2
at2.sinks.kt2.channel = ct2

在该文件所存的目录下，执行以下命令调起：

flume-ng agent -name at2  -conf-file test-kafka-to-file.properties

注：sink.rollInterval=300，表示每隔300秒分割成一个文件，默认为30秒，如果不想分割，可以写为0，数据全部追加在一个文件中。