(原)logstash-forwarder + logstash + elasticsearch + kibana

[logstash-forwarder + logstash + elasticsearch + kibana]
------------------------------------------------------------------------------------------------------------------------------------------------
摘要:logstash-forwarder搜集日志，汇总给logstash，然后输出到elasticsearch,并由kibana展现web界面.
------------------------------------------------------------------------------------------------------------------------------------------------
一 安装
1.logstash-forwarder
see and install:
https://github.com/elasticsearch/logstash-forwarder

(logstash-forwarder有个坑. 虽然严格讲不算是logstash-forwarder的坑.
跟证书相关的:https://github.com/elasticsearch/logstash-forwarder/issues/221 <-可以不看.
下面的解决方案规避这个坑了. 下面会提到.)

2.logstash
see and install:
http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash

3.elasticsearch

3.1.下载https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz

3.2.解压到目录 elasticsearch-1.3.2

3.3. 测试安装是否成功
$ cd elasticsearch-1.3.2/
$ bin/elasticsearch
$ curl -X GET http://localhost:9200/
(保持elasticsearch一直运行. 下面将继续测试)

4.kibana:

4.1.下载https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz

4.2. 解压到目录 kibana-3.1.0

4.3. 测试安装是否成功
$ cd kibana-3.1.0
$ vi config.js
第32行修改为：
elasticsearch: "http://localhost:9200",
或者如果是要非本地访问,就应该这样:
elasticsearch: "http://"+window.location.hostname+":9200"
注意后面有逗号.
在浏览器里打开这目录里的index.html.

------------------------------------------------------------------------------------------------------------------------------------------------
二 .方案:

client[logstash-forwarder]---|
client[logstash-forwarder]---|---log-server[logstash]--->[elasticsearch]
client[logstash-forwarder]---|

2.1 先启动elasticsearch
前面已经启动了.

2.2  开启logstash
先写logstash的配置文件:
$ cd logstash-1.4.2
$ vi test_logstash.conf
input {
  lumberjack {
    # The port to listen on
    port => 5000

# The paths to your ssl cert and key
    ssl_certificate => "/home/xiaou/logstash-forwarder.crt"
    ssl_key => "/home/xiaou/logstash-forwarder.key"

# Set this to whatever you want.
    type => "somelogsXXX"
  }
}
output {
  elasticsearch { host => localhost } # 因为logstash和elasticsearch在同一台机器上，所以这里可以用localhost
  stdout { codec => rubydebug }
}

还要产生自签证书:
$ openssl req -subj '/CN=localhost/' -x509 -batch -nodes -newkey rsa:2048 -keyout /home/xiaou/logstash-forwarder.key -out /home/xiaou/logstash-forwarder.crt  -days 1095
(这里用“-subj '/CN=localhost/'”规避了上面提到的logstash-forwarder的坑)

然后启动logstash：
$ bin/logstash -f test_logstash.conf

2.3 启动logstash-forwarder
先写logstash-forwarder的配置文件:
$ cd logstash-forwarder
$ vi test_forwarder.conf
{
  "network": {
    "servers": [ "localhost:5000" ],
 "ssl ca": "/home/xiaou/logstash-forwarder.crt",
    "timeout": 5
  },
  "files": [
    {
      "paths": [
        "/var/log/linshi.txt",
        "/var/log/*.log"
      ],
      "fields": {
        "type": "linshiXX"
      }
    }
  ]
}
(这里配置文件的写法也是规避了前面提到的logstash-forwarder的坑:servers没用ip)
启动logstash-forwarder:
$ ./logstash-forwarder -config test_forwarder.conf
logstash-forwarder启动后就会与logstash建立tcp连接.

测试, 写日志，观察运行logstash的终端的输出:
$ echo 1234 >> /var/log/linshi.txt

2.4  打开kibana,展现最终汇总到elasticsearch的日志.
(唯kibana不能算是服务, 它只是一个“阅读器”.)
用浏览器打开kibana-3.1.0目录下的index.html,看右边倒数第五行有个链接。打开.
------------------------------------------------------------------------------------------------------------------------------------------------
三.深入：

1. type
logstash.conf里的
input {
  lumberjack {
    ...
    type => "this forwarder's file have no type!"
这个type,是对forwarder.conf的补充:如果forwarder.conf里没有type，则这里的type就会填充日志event的type字段.
ps:
一条日志event是这样的:
{
       "message" => "xx",
      "@version" => "1",
    "@timestamp" => "2014-09-18T03:31:12.744Z",
          "type" => "linshi1",
          "file" => "/var/log/epoch/linshi.txt",
          "host" => "xiaou-mint",
        "offset" => "568"
}
用type来作为区分各个日志应该不错:
在forwarder里这样写files:
  "files": [
    {
      "paths": [
        "/var/log/epoch/linshi1.txt"
      ],
      "fields": {
        "type": "linshi1"
      }
    },
    {
      "paths": [
        "/var/log/epoch/linshi2.txt"
      ],
      "fields": {
        "type": "linshi2"
      }
    }
  ]

2.add_field添加字段
    add_field => {
      "test_field" => "asdasd"
      "test_filed2" => "112233"
    }
尽量不要跟日志event里已有的字段冲突了，如果要这么做，需要自行测试是否会覆盖event日志的字段. 我测试了几个字段诸如type、message、file，居然表现各一,无法统一下结论.

3.if表达式
随时需要查文档http://logstash.net/docs/1.4.2/。。。不写了. End.
/*
http://logstash.net/docs/1.4.2/inputs/lumberjack
http://logstash.net/docs/1.4.2/configuration#conditionals
http://logstash.net/docs/1.4.2/filters/mutate
http://logstash.net/docs/1.4.2/filters/drop
*/

4. 最后给出两个conf的测试内容:
logstash.conf :

input {
  lumberjack {
    # The port to listen on
    port => 5000

    # The paths to your ssl cert and key
    ssl_certificate => "/home/xiaou/logstash-forwarder.crt"
    ssl_key => "/home/xiaou/logstash-forwarder.key"

    type => "this forwarder's file have no type!"

  }
}

filter{
  if [type] == "linshi2"{
    mutate{
      replace => ["message","%{message}:it's linshi2"]
      update => ["file", "FILE_LINSHI2"] # 替换字段.
    }
  }else{ # linshi1
    if "error" in [message]{ # 日志里还有“error”字符串
      mutate{
        add_field => {"NOTE" => "ERROR!"} # 添加字段
        add_tag => "tag_error!" # 添加标签. 标签是个数组
        add_tag => "tag_error2!"
      }
    }else{ # 如果来自linshi1.txt的并且没有“error”自负, 则丢弃.
      drop{}
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

forwarder.conf :

{
  "network": {
    "servers": [ "localhost:5000" ],
 "ssl ca": "/home/xiaou/logstash-forwarder.crt",
    "timeout": 5
  },

  "files": [
    {
      "paths": [
        "/var/log/epoch/linshi1.txt"
      ],
      "fields": {
        "type": "linshi1"
      }
    },
    {
      "paths": [
        "/var/log/epoch/linshi2.txt"
      ],
      "fields": {
        "type": "linshi2"
      }
    }
  ]
}

------------------------------------------------------------------------------------------------------------------------------------------------
End.