CentOS6.9安装Filebeat监控Nginx的访问日志发送到Kafka

一、下载地址：

官方：https://www.elastic.co/cn/downloads/beats/filebeat

百度云盘：https://pan.baidu.com/s/1dvhqb0

二、安装

tar zxvf filebeat-6.2.-linux-x86_64.tar.gz
mv filebeat-6.2.-linux-x86_64 filebeat

也可yum安装

下载和安装key文件

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

创建yum源文件

vi /etc/yum.repos.d/elastic.repo

[elastic-.x]
name=Elastic repository for .x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=
autorefresh=
type=rpm-md

开始安装

yum install filebeat

三、配置

vi kafka.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /usr/local/openresty/nginx/logs/access.log
output.kafka:
  hosts: ["10.10.6.225:9092"]
  topic: need_process_nginx_access_log
  partition.round_robin:
    reachable_only: false
  required_acks: 

注：output到kafka的partitions策略：分为random、round_robin、hash，默认是hash

具体说明：https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#_literal_partition_literal

运行

./filebeat -e -c kafka.yml

后台运行

nohup /usr/local/filebeat/filebeat -e -c /usr/local/filebeat/kafka.yml >/dev/null 2>&1 &