logstash输出到elasticsearch多索引

目标：将json格式的两类日志输出到elasticsearch两类索引

1. 安装logstash。

2. 编写logstash处理配置文件，创建一个test.conf文件，内容如下：

input {
     file {
        path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-*.log"
        start_position => "beginning"
        type => "sql"
        codec => json {
            charset => "UTF-8"
        }
     }
     file {
        path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-transaction-*.log"
        start_position => "beginning"
        type => "transaction"
        codec => json {
            charset => "UTF-8"
        }
     }
}
output {
    if "_grokparsefailure" in [tags] {
    }else{
        if [type] == "sql"{
                elasticsearch {
                         hosts => ["http://192.168.33.10:9200"]
                         index => "common-sql-%{+YYYY.MM.dd}"
                }
       }
       if [type] == "transaction"{
                elasticsearch {
                         hosts => ["http://192.168.33.10:9200"]
                         index => "common-transaction-%{+YYYY.MM.dd}"
                }
       }
    }
}

或者

input {
     file {
        path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-sql-*.log"
        start_position => "beginning"
        type => "sql"
        codec => json {
            charset => "UTF-8"
        }
     }
     file {
        path => "/home/vagrant/logstash/logstash-2.2.2/dbpool-logs/dev/common-transaction-*.log"
        start_position => "beginning"
        type => "transaction"
        codec => json {
            charset => "UTF-8"
        }
     }
}
output {
    if "_grokparsefailure" in [tags] {
    }else{
        if [type] == "sql"{
                elasticsearch {
                         hosts => ["http://192.168.33.10:9200"]
                         index => "common-%{type}-%{+YYYY.MM.dd}"
                }
       }
    }
}

• 指定输入日志的路径，按通配符匹配。分为两类：sql和transaction。

• 根据type分别输出到elasticsearch不同的索引。

3. 安装elasticsearch。

4. 启动elasticsearch，./bin/elasticsearch -d ，默认端口为9200。

5. 启动logstash开始处理，./bin/logstash -f conf/test.conf。

6. 完成。