ELK Kafka json to elk

Logstash配置

 
 

input {

kafka {

zk_connect => "127.0.0.1:2181"

topic_id => "cluster"

codec => plain

reset_beginning => false

consumer_threads => 5

decorate_events => true

}

}

 
 

 
 

output {

if [type]=="cluster3" or [type]=="cluster2" or [type]=="clusterjson"

{

elasticsearch {

hosts => ["localhost:9200"]

index => "test-kafka-%{type}-%{+YYYY-MM}"

}

}

 
 

stdout { codec => rubydebug }

}

 
 

Server.properties 主要内容

broker.id=0

 
 

############################# Socket Server Settings #############################

 
 

listeners=PLAINTEXT://:9092

 
 

# The port the socket server listens on

port=9092

 
 

 
 

Server-1.properties 主要内容

 
 

broker.id=1

 
 

############################# Socket Server Settings #############################

 
 

listeners=PLAINTEXT://:9093

 
 

# The port the socket server listens on

port=9093

 
 

 
 

启动kafka,server.properties和server-1.properties

E:\soft\elk\kafka_2.11-0.9.0.1>bin\windows\kafka-server-start.bat config\custom\server-1.properties

 
 

创建topic,并查看状态

 
 

 
 

查找端口占用情况，并删除一个节点

 
 

 
 

查看 topic状态

 
 

 
 

 
 

生产者输入json数据，如下

 
 

{"Name":"BULK 7","Data":7,"CreateTime":"2016-04-05T10:16:22Z","type":"cluster3"}

 
 

Kibana 4显示如下