ELKBR部署检测项目日志

ELK

• filebeat：具有日志收集功能，相比logstash，+filebeat更轻量，占用资源更少，适合客户端使用。
• redis消息队列选型：Redis 服务器通常都是用作 NoSQL 数据库，不过这里的 redis 只是用来做消息队列，海量日志建议使用kafka。
• logstash：主要是用来日志的搜集、分析、过滤日志的工具，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。
• elasticsearch：Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
• kibana：Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。

主机名	配置	角色	软件版本
filebeat	无	日志搜集客户端	https://www.elastic.co/cn/downloads/beats/filebeat
redis	2 vCPU 8 GiB	消息队列	redis-stable.tar.gz
logstash	4 vCPU 16 GiB	日志搜集	https://www.elastic.co/cn/downloads/logstash
elasticsearch	4 vCPU 16 GiB	日志搜索引擎	https://www.elastic.co/cn/downloads/elasticsearch
kibana	2 vCPU 4 GiB	日志数据展示	https://www.elastic.co/cn/downloads/kibana

ES激活白金版：https://www.jianshu.com/p/1ff67bb363dd

1 redis部署

• 上传部署包至/usr/local/src/redis/，依赖安装及环境配置

• mkdir -p /usr/local/src/redis/
  yum -y install gcc-c++ make tcl
  service iptables stop && chkconfig iptables off
  echo 511 > /proc/sys/net/core/somaxconn
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  echo "echo 511 > /proc/sys/net/core/somaxconn" >> /etc/rc.local
  echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
  echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf && modprobe bridge && sysctl -p
  

• 编译安装redis server

• mkdir -p /usr/local/src/redis/{data,log}
  cd /usr/local/src/redis && tar -zxvf redis-stable.tar.gz -C . && cd redis-stable && make && make install
  (此处可一路回车选择默认配置,配置文件在cat /etc/redis/6379.conf)
  ./utils/install_server.sh
  

• 启动redis

• chkconfig redis_6379 on
  service redis_6379 start
  

• 检查启动后服务是否正常

• redis-cli -h 本机IP
  

  **localhost:6379> keys ***
  
  (empty list or set)
  
  localhost:6379> set test abc
  
  OK
  
  localhost:6379> get test
  
  "abc"
  
  localhost:6379> del test
  
  (integer) 1

  localhost:6379> quit

2 filebeat

• 上传filebeat包，解压修改配置

• mkdir /usr/local/src/filebeat
  #上传压缩包
  tar -zxvf filebeat-7.3.1-linux-x86_64.tar.gz
  cd filebeat-7.3.1-linux-x86_64
  vim filebeat.yml
  

(paths:对应日志位置)

enabled: true

paths:

• /tomcat/apache-tomcat-7.0.72/logs/*.out

multiline.pattern: ^[

multiline.negate: false

multiline.match: after

(tags:变量TEMPLATE_TAG用于部署到对应服务器后批量替换为对应实例名称)

tags: ["192.168.192.10"]

注释掉Elasticsearch相关的配置

----------------------------- Redis output --------------------------------

(key变量为保存到redis中的键名，用于部署到对应服务器后批量替换为对应应用名)

output.redis:

hosts: ["localhost:6379"]

db: 0

timeout: 5

key: "project-name"

================================ Global =====================================

filebeat.global:

filebeat.spool_size: 64

filebeat.idle_timeout: 5s

• 启动

• ./filebeat -e -c filebeat.yml
  

• 查看redis是否已有日志记录

• redis-cli -h 本机IP
  #查看所有KEY
  keys *
  #查看日志长度
  LLEN project-name
  #查看某一行日志
  INDEX project-name 1
  
  

3 ElasticSearch

• 下载解压

• mkdir /usr/local/src/elasticsearch
  #上传压缩包
  tar -zxvf elasticsearch-7.3.1-linux-x86_64.tar.gz
  

• 在es下创建数据和日志文件夹，修改配置文件

• #对应配置里的数据和日志文件目录
  mkdir /usr/local/src/elasticsearch/elasticsearch-7.3.1/data
  #mkdir /usr/local/src/elasticsearch/elasticsearch-7.3.1/logs
  vim config/elasticsearch.yml
  

  cluster.name: my-application

  node.name: node-10

  path.data: /usr/local/src/elasticsearch/elasticsearch-7.3.1/data

  path.logs: /usr/local/src/elasticsearch/elasticsearch-7.3.1/logs

  network.host: 本机IP
  
  http.port: 9200
  
  discovery.seed_hosts: ["本机IP"]
  
  cluster.initial_master_nodes: ["node-10"]

  因为Centos6不支持SecComp加入下面两句

  bootstrap.memory_lock: false
  
  bootstrap.system_call_filter: false

  验证开启(不想要密码就忽略)

  xpack.security.enabled: true
  
  xpack.ml.enabled: true
  
  xpack.license.self_generated.type: trial

• 新建elkuser用户(注意不能使用root用户启动es)

• #创建用户
  useradd elkuser
  #修改用户组和用户
  cd /usr/local/src/elasticsearch
  chown -R elkuser:elkuser elasticsearch-7.3.1
  #切换用户
  su elkuser
  

• 启动

• ./bin/elasticsearch
  

• 设置启动密码

• ./bin/elasticsearch-setup-passwords interactive --verbose
  

• Trying user password change call http://192.168.192.10:9200/_security/user/apm_system/_password?pretty
  
  { }

  Changed password for user [apm_system]

  Trying user password change call http://192.168.192.10:9200/_security/user/kibana/_password?pretty
  
  { }

  Changed password for user [kibana]

  Trying user password change call http://192.168.192.10:9200/_security/user/logstash_system/_password?pretty
  
  { }

  Changed password for user [logstash_system]

  Trying user password change call http://192.168.192.10:9200/_security/user/beats_system/_password?pretty
  
  { }

  Changed password for user [beats_system]

  Trying user password change call http://192.168.192.10:9200/_security/user/remote_monitoring_user/_password?pretty
  
  { }

  Changed password for user [remote_monitoring_user]

  Trying user password change call http://192.168.192.10:9200/_security/user/elastic/_password?pretty
  
  { }

  Changed password for user [elastic]

  • 登陆 http://192.168.192.10:9200/ 用户名:elastic 密码(刚刚设置的): elkuser
  • 

ElasticSearch启动问题:

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

切换到root用户，编辑limits.conf添加如下内容(需要重启生效)

vi /etc/security/limits.conf

* soft nofile 65536

* hard nofile 65536

[2]: max number of threads [3818] for user [es] is too low, increase to at least [4096]

最大线程个数太低。修改配置文件etc/security/limits.conf，增加配置

* soft nproc 4096

* hard nproc 4096

[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

修改/etc/sysctl.conf文件，增加配置vm.max_map_count=262144

vi /etc/sysctl.conf

sysctl -p

执行命令sysctl -p生效

[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

问题原因：因为Centos6不支持SecComp，而ES5.2.1默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动

解决方法：在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:

bootstrap.memory_lock: false

bootstrap.system_call_filter: false

4 Logstash

• 下载解压

• mkdir /usr/local/src/logstash
  #上传压缩包
  tar -zxvf logstash-7.3.1.tar.gz
  

• 创建一个配置文件

• vim logstash.conf
  

  input {

  ​ redis {
  
  ​ host => "redis所在机器"
  
  ​ data_type => "list"
  
  ​ port => "6379"
  
  ​ key => "project-name"
  
  ​ type => "project-name"
  
  ​ }

  }

  output {

  ​ if[type == "project-name"]{

  ​ elasticsearch {
  
  ​ hosts => "es所在机器:9200"
  
  ​ user => "elastic"
  
  ​ password => "刚刚es环境设置的密码"
  
  ​ codec => "json"
  
  ​ index => "project-name-%{+YYYY.MM.dd}"
  
  ​ }

  ​ }

  }

• 启动

• bin/logstash -f logstash.conf
  

5 kibana

• 下载解压

• mkdir /usr/local/src/kibana
  #上传压缩包
  tar -zxvf kibana-7.3.1-linux-x86_64.tar.gz
  
  cd kibana-7.3.1-linux-x86_64
  mkdir logs
  #进入kibana配置下
  cd config
  #修改配置文件
  vim kibana.yml
  

server.port: 5601

server.host: "本机IP"

elasticsearch.hosts: ["http://ES服务器IP:9200"]

kibana.index: ".kibana"

elasticsearch.username: "kibana"

elasticsearch.password: "es环境设置的kibana密码"

pid.file: /var/run/kibana.pid

logging.dest: /usr/local/src/kibana/kibana-7.3.1-linux-x86_64/logs/kibana.log

• 启动(kibana不建议以root用户启动，如果用root启动，需要加--allow-root)

• bin/kibana
  #bin/kibana  --allow-root
  (观察配置的日志)
  

• 访问5601端口(使用es用户)(以下没有更新，参考使用)

• 

• 创建日志索引

• 

• 

• 

• 查看日志量等信息

•