05 . ELK Stack+Redis日志收集平台

环境清单

IP	hostname	软件	配置要求	网络	备注
192.168.43.176	ES/数据存储	elasticsearch-7.2	内存2GB/硬盘40GB	Nat，内网
192.168.43.215	Kibana/UI展示	kibana-7.2	内存2GB/硬盘40GB	Nat,内网
192.168.43.164	Filebeat/数据采集	Filebeat-7.2/nginx	内存2GB/硬盘40GB	Nat，内网
192.168.43.86	Kibana/UI展示	kibana-7.2	内存2GB/硬盘40GB	Nat，内网
192.168.43.30	Logstash/数据管道	logstash-7.2	内存2GB/硬盘40GB	Nat，内网
192.168.43.47	Redis/消息队列	Redis-4.0	内存2GB/硬盘40GB	Nat，内网
192.168.43.205	nginx	nginx1.14	内存2GB/硬盘40GB	Nat，内网

配置日志采集端Nginx

修改Nginx日志格式

nginx日志默认格式为log格式，传输到es中需要经过grok插件进行处理并转换成json格式，这一过程是很消耗logstash资源的，而且传入到es中的字段并不容易分析，所以在收集端先将日志转为json格式，再传入es中去，这样传入的字段也是利于分析的。

编辑nginx配置文件

log_format json '{ "@timestamp": "$time_iso8601", '
                         '"time": "$time_iso8601", '
                         '"clientip": "$remote_addr", '
                         '"remote_user": "$remote_user", '
                         '"body_bytes_sent": "$body_bytes_sent", '
                         '"request_time": "$request_time", '
                         '"status": "$status", '
                         '"host": "$host", '
                         '"request": "$request", '
                         '"request_method": "$request_method", '
                         '"uri": "$uri", '
                         '"http_referrer": "$http_referer", '
                         '"body_bytes_sent":"$body_bytes_sent", '
                         '"http_x_forwarded_for": "$http_x_forwarded_for", '
                         '"http_user_agent": "$http_user_agent" '
                    '}';

    access_log  /var/log/nginx/access.log  json;

重启Nginx

systemctl restart nginx

访问产生日志

curl localhost -I

验证数据

tailf /var/log/nginx/access.log
{ "@timestamp": "2020-07-21T19:54:27+08:00", "time": "2020-07-21T19:54:27+08:00", "clientip": "192.168.43.45", "remote_user": "-", "body_bytes_sent": "0", "request_time": "0.000", "status": "304", "host": "192.168.43.205", "request": "GET / HTTP/1.1", "request_method": "GET", "uri": "/index.html", "http_referrer": "-", "body_bytes_sent":"0", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36" }
{ "@timestamp": "2020-07-21T21:53:54+08:00", "time": "2020-07-21T21:53:54+08:00", "clientip": "127.0.0.1", "remote_user": "-", "body_bytes_sent": "0", "request_time": "0.000", "status": "200", "host": "localhost", "request": "HEAD / HTTP/1.1", "request_method": "HEAD", "uri": "/index.html", "http_referrer": "-", "body_bytes_sent":"0", "http_x_forwarded_for": "-", "http_user_agent": "curl/7.29.0" }

配置日志采集端Redis

Redis消息队列使用说明

redis服务器是logstash官方推荐的broker（代理人）选择，broker角色也就意味着会同时存在输入和输出两个插件，产生数据的被称作生产者，而消费数据的被称作消费者。

# 1、防止Logstash和ES无法正常通信，从而丢失日志。
# 2、防止日志量过大导致ES无法承受大量写操作从而丢失日志。
# 3、应用程序（php，java）在输出日志时，可以直接输出到消息队列，从而完成日志收集。
# 补充：如果redis使用的消息队列出现扩展瓶颈，可以使用更加强大的kafka，flume来代替。

编译安装Redis

wget http://download.redis.io/releases/redis-4.0.11.tar.gz  #下载Redis源码
tar -zxf redis-4.0.11.tar.gz  -C /usr/local    #解压Redis源码
make && make install  PREFIX=/usr/local/redis    #编译安装Redis
echo "export PATH=$PATH:/usr/local/redis/bin" >> /etc/profile  # 将Redis加入环境变量

启动Redis

# 1.前端模式启动
# 直接运行bin/redis-server将以前端模式启动，前端模式启动的缺点是ssh命令窗口关闭则redis-server程序结束，不推荐使用此方法启动redis-server
# 2.后端模式启动
# 修改redis.conf配置文件， daemonize yes 以后端模式启动
vim /usr/local/redis/bin/redis.conf
daemonize yes
redis-server redis.conf

连接Redis和关闭Redis

# 连接redis
redis-cli

# 强行终止redis进程可能会导致redis持久化数据丢失。正确停止Redis的方式应该是向Redis发送SHUTDOWN命令，命令为：
redis-cli shutdown

安装配置Filebeat

filebeat是一个轻量级的日志采集器，由于logstash比较消耗资源，不适合在每台主机上部署logstash

rpm安装FIlebeat

rpm -vi filebeat-7.2.0-x86_64.rpm

配置filebeat收集nginx日志

配置输入端采集nginx日志，根据字段类型不同输出到Redis不同的key中，每种日志存放在不同的key中，便于后续的处理

 cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access*.log
  fields:
    type: www_access
  fields_under_root: true
- type: log
  paths:
    - /var/log/nginx/error*.log
  fields:
    type: www_error
  fields_under_root: true
- type: log
  paths:
    - /var/log/nginx/doc.access.log
  fields:
    type: doc_access
  fields_under_root: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.redis:
  hosts: ["192.168.43.47:6379"]
  key: "nginx"
  keys:
    - key: "%{[type]}"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Redis端验证数据

[root@redis ~]# redis-cli
127.0.0.1:6379> keys *
1) "a"
2) "www_access"
3) "www_error"
127.0.0.1:6379> LINDEX www_access 3
"{\"@timestamp\":\"2020-07-22T06:26:49.662Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"7.2.0\"},\"input\":{\"type\":\"log\"},\"type\":\"www_access\",\"ecs\":{\"version\":\"1.0.0\"},\"host\":{\"hostname\":\"nginx\",\"architecture\":\"x86_64\",\"os\":{\"name\":\"CentOS Linux\",\"kernel\":\"3.10.0-514.el7.x86_64\",\"codename\":\"Core\",\"platform\":\"centos\",\"version\":\"7 (Core)\",\"family\":\"redhat\"},\"id\":\"b029c3ce28374f7db698c050e342457f\",\"containerized\":false,\"name\":\"nginx\"},\"agent\":{\"ephemeral_id\":\"76f6177a-dd5b-4c40-9b61-8f406507c6cb\",\"hostname\":\"nginx\",\"id\":\"cf47c715-17f2-48d5-9f10-866f10eba0cf\",\"version\":\"7.2.0\",\"type\":\"filebeat\"},\"log\":{\"file\":{\"path\":\"/var/log/nginx/access.log\"},\"offset\":8995},\"message\":\"192.168.43.45 [21/Jul/2020:19:20:16 +0800] \\\"GET / HTTP/1.1\\\" 304 0 \\\"-\\\" 0.000 \\\"-\\\" - \\\"Mozilla/5.0 (Windows NT 10.0

配置日志处理端Logstash

安装并配置logstash

rpm -ivh jdk-8u121-linux-x64.rpm
tar xvf logstash-7.2.0.tar.gz  -C /opt/

编写日志处理配置文件

定义Redis列表或者频道名称，以及Redis的数据类型，定义type以区分不同的日志类型，使用json插件将message字段处理成json格式，并删掉message字段，使用date插件定义新的时间戳，使用geoip插件根据客户端IP来定位客户端大体，默认是使用GeoLite2 city数据库，此数据库官网每两周更新一次，如果对IP地址的准确性要求高，可写一个定时任务，每两周从官网下载新的数据库，mutate插件用于修改字段数据类型，因为”coordinates”子字段不修改会默认为keyword格式，这对于在kibana上创建坐标地图可能会不支持.

test.conf

cat /opt/logstash-7.2.0/test.conf
input {
    redis {
        host => "192.168.43.47"
        port => 6379
        type => "www_access"
        data_type => "list"
        key => "www_access"
        codec => "json"
    }
    redis {
        host => "192.168.43.47"
        port => 6379
        type => "nginx_error"
        data_type => "list"
        key => "www_error"
    }
    redis {
        host => "192.168.43.47"
                port => 6379
                type => "doc_access"
                data_type => "list"
                key => "doc_access"
                codec => "json"
    }
}
filter {
    if [type] == "www_access" {
            json {
               source => "message"
               remove_field => "message"
             }
            date {
              match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
        geoip {
          source => "clientip"
          fields => ["city_name", "country_code2", "country_name", "region_name","longitude","latitude","ip"]
              add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
              add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
              convert => [ "[geoip][coordinates]","float" ]
        }
    }
        else if [type] =~ "error" {
            grok {
            }
            date {
              match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
            }
        }
        else {
            json {
               source => "message"
                   remove_field => "message"
            }
            date {
              match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]              }
            geoip {
                  source => "clientip"
                  fields => ["city_name", "country_code2", "country_name", "region_name","longitude","latitude","ip"]
               add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                  add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]                }
            mutate {
            convert => ["[geoip][coordinates]","float"]
           }
        }

}
output {
        if [type] == "www_access" {
      elasticsearch {
        hosts => ["192.168.43.176:9200"]
        index => "nginx_access-%{+YYYY.MM.dd}"
      }
      stdout {
                codec => rubydebug
          }
    }
    else if [type] == "www_error" {
      elasticsearch {
        hosts => ["192.168.43.215:9200"]
        index => "nginx_error-%{+YYYY.MM.dd}"
      }
    }
    else {
      elasticsearch {
        hosts => ["192.168.43.164:9200"]
                index => "nginx_doc-%{+YYYY.MM.dd}"
      }
      stdout {
        codec => rubydebug
          }
    }
}

启动Logstash

./bin/logstash   -f test.conf
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /opt/logstash-7.2.0/logs which is now configured via log4j2.properties
[2020-07-22T15:26:51,015][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-07-22T15:26:51,110][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2020-07-22T15:27:02,202][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.43.176:9200/]}}

验证Redis数据是否被消费

# 我们可以看到日志都已经被消费了
127.0.0.1:6379> LLEN www_access
(integer) 0
127.0.0.1:6379>  LINDEX www_access 1
(nil)

配置日志存储分析端

安装elasticsearch集群和kibana此处不做介绍,请看我之前文章

https://www.cnblogs.com/you-men/p/12761738.html

https://www.cnblogs.com/you-men/p/13167801.html

验证传递过来的nginx索引

[root@es1 ~]#  curl -XGET "http://127.0.0.1:9200/_cat/indices?v"
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                         6LXFwoifQFWS7JUbWsSmkw   1   1          3            0     29.1kb         14.5kb
green  open   .monitoring-es-7-2020.07.22     SPa4aMKzRDSiv9EoUj-dkw   1   1       8707         7870     27.5mb          8.6mb
green  open   nginx_error-2020.07.22          8-ka5wT2T962BSmm1IkUoQ   1   1          7            0     44.3kb         22.1kb
green  open   .monitoring-es-7-2020.07.21     XmwuVQkyQfeyR2Jq6J4EmA   1   1      36700        18108       35mb         16.8mb
green  open   .monitoring-kibana-7-2020.07.22 fdi9l1vLRFOsVKoMeImlUQ   1   1       1337            0      744kb        372.5kb
green  open   nginx_access-2020.07.22         bN-5RWhBTfW4rPgTSPIJNg   1   1          9            0     46.6kb         23.3kb
green  open   nginx_access-2020.07.21         cO7Us2FRR8KAhgp2AP228g   1   1          5            0     39.4kb         19.7kb

green  open   .monitoring-kibana-7-2020.07.21 hGIIp9KST0Kq60owKMWLmg   1   1       4080            0      1.7mb        887.2kb

配置kibana

也可以配置grafana集成到一个平台上