基本环境：
filebeat版本：6.5.4 (Linux,x86-64)
elasticsearch版本：6.54
 
（一）需求说明
在一台服务器上有多个日志需要使用filebeat日志收集到elasticsearch中，以便于查看。对于收集方法，主要有2种：
  • 将同一台服务器上的日志收集到elasticsearch的同一个索引中，这种方式存在一个较大的问题，如果服务器上有多个业务在运行，产生了多个日志，那么将会被收集到elasticsearch的同一个索引中，如图1。
  • 将同一台服务器上的日志收集到elasticsearch的不同索引中，每个索引都存放相关业务的日志，如图2。
 
 
很明显，图2的日志输出是我们想要的，因为它将不同的日志放到了不同的索引中。
 
（二）解决方案
在使用filbeat收集日志输出到elasticsearch数据库时，可以使用indices参数来配置不同的日志输出到不同的索引中，官方文档及其配置例子如下：
 
 
（三）实际测试
（3.1）输入存在fields_under_root: true选项
使用filebeat对3个日志testa.log、testb.log、testc.log进行数据抓取，要求：
  • testa.log日志的数据存放到testa-log索引中
  • testb.log日志的数据存放到testb-log索引中
  • 其它(非testa.log和testb.log)的日志数据存放到test-other-log索引中
 
filebeat输入配置如下：
 
输出配置如下：
 
 
最终测试成功。
 
这里附一份完整的filebeat配置文件：

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

# testa.log

- type: log

  enabled: true

  paths:

    - /root/test/testa.log

  fields:

    log_topics: "testa"

  fields_under_root: true

# testb.log

- type: log

  enabled: true

  paths:

    - /root/test/testb.log

  fields:

    log_topics: "testb"

  fields_under_root: true

# testc.log

- type: log

  enabled: true

  paths:

    - /root/test/testc.log

  fields:

    log_topics: "testc"

  fields_under_root: true

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  #  level: debug

  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

  reload.enabled: true

  # Period on which files under path should be checked for changes

  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:

  index.number_of_shards: 1

  #index.codec: best_compression

  #_source.enabled: false

setup.template.name: "prod-file*"

setup.template.pattern: "prod-file*"

setup.ilm.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

#============================== Dashboards =====================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the artifacts.elastic.co

# website.

#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  #host: "localhost:5601"

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  index: "testlog-666"

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  indices:

#    - index: "testa-log"

#      when.contains:

#        log_topics: "testa"

#    - index: "testb-log"

#      when.contains:

#        log_topics: "testb"

output.elasticsearch:

  hosts: ["192.168.10.100:9200"]

  index: "test-other-log"

  indices:

    - index: "testa-log"

      when.contains:

        log_topics: "testa"

    - index: "testb-log"

      when.contains:

        log_topics: "testb"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================

# filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true
 （3.2）输入不存在fields_under_root: true选项
filebeat输入配置如下：
 
输出配置如下：
 
这里附一份完整的filebeat配置文件：

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

# testa.log

- type: log

  enabled: true

  paths:

    - /root/test/testa.log

  fields:

    log_topics: "testa"

# testb.log

- type: log

  enabled: true

  paths:

    - /root/test/testb.log

  fields:

    log_topics: "testb"

# testc.log

- type: log

  enabled: true

  paths:

    - /root/test/testc.log

  fields:

    log_topics: "testc"

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  #  level: debug

  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

  reload.enabled: true

  # Period on which files under path should be checked for changes

  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:

  index.number_of_shards: 1

  #index.codec: best_compression

  #_source.enabled: false

setup.template.name: "prod-file*"

setup.template.pattern: "prod-file*"

setup.ilm.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

#============================== Dashboards =====================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the artifacts.elastic.co

# website.

#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  #host: "localhost:5601"

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  index: "testlog-666"

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  indices:

#    - index: "testa-log"

#      when.contains:

#        log_topics: "testa"

#    - index: "testb-log"

#      when.contains:

#        log_topics: "testb"

output.elasticsearch:

  hosts: ["192.168.10.100:9200"]

  index: "test-other-log"

  indices:

    - index: "testa-log"

      when.contains:

        fields:

          log_topics: "testa"

    - index: "testb-log"

      when.contains:

        fields:

          log_topics: "testb"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================

# filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true
 
 
 其它说明：为什么需要特别注意fields_under_root参数
fields_under_root参数定义如下：
  • 如果值为ture，那么fields存储在输出文档的顶级位置，如果与filebeat中字段冲突，自定义字段会覆盖其他字段
  • 如果值为false或者未设置，那么fields存储在输出文档的子位置。
如下：
（1）fields_under_root：true
此时在filebeat的input部分定义的字段log_topics是一个顶级字段。
 
 
 
 
（2）fields_under_root：false或者未设置
此时在filebeat的input部分定义的字段log_topics是fields字段的子字段。
 
 
 
 
【完】

filebeat输出结果到elasticsearch的多个索引的更多相关文章

  1. 使用ElasticSearch赋能HBase二级索引 | 实践一年后总结

    前言:还记得那是2018年的一个夏天,天气特别热,我一边擦汗一边听领导大刀阔斧的讲述自己未来的改革蓝图.会议开完了,核心思想就是:我们要搞一个数据大池子,要把公司能灌的数据都灌入这个大池子,然后让别人 ...

  2. 第三百六十二节，Python分布式爬虫打造搜索引擎Scrapy精讲—elasticsearch(搜索引擎)基本的索引和文档CRUD操作、增、删、改、查

    第三百六十二节,Python分布式爬虫打造搜索引擎Scrapy精讲—elasticsearch(搜索引擎)基本的索引和文档CRUD操作.增.删.改.查 elasticsearch(搜索引擎)基本的索引 ...

  3. (转)ElasticSearch Java Api-检索索引库

    上篇博客记录了如何用java调用api把数据写入索引,这次记录下如何搜索. 一.准备数据 String data1 = JsonUtil.model2Json(new Blog(1, "gi ...

  4. 四十一 Python分布式爬虫打造搜索引擎Scrapy精讲—elasticsearch(搜索引擎)基本的索引和文档CRUD操作、增、删、改、查

    elasticsearch(搜索引擎)基本的索引和文档CRUD操作 也就是基本的索引和文档.增.删.改.查.操作 注意:以下操作都是在kibana里操作的 elasticsearch(搜索引擎)都是基 ...

  5. Elasticsearch之curl创建索引

    前提,是 Elasticsearch之curl创建索引库 [hadoop@djt002 elasticsearch-2.4.3]$ curl -XPUT 'http://192.168.80.200: ...

  6. Elasticsearch之curl创建索引库

    关于curl的介绍,请移步 Elasticsearch学习概念之curl 启动es,请移步 Elasticsearch的前后台运行与停止(tar包方式) Elasticsearch的前后台运行与停止( ...

  7. Elasticsearch之curl删除索引库

    关于curl创建索引库的介绍,请移步 Elasticsearch之curl创建索引库 [hadoop@djt002 elasticsearch-2.4.3]$ curl -XPUT 'http://1 ...

  8. Elasticsearch之curl创建索引库和索引时注意事项

    前提, Elasticsearch之curl创建索引库 Elasticsearch之curl创建索引 注意事项 1.索引库名称必须要全部小写,不能以下划线开头,也不能包含逗号 2.如果没有明确指定索引 ...

  9. Elasticsearch之cur查询索引

    前提, Elasticsearch之curl创建索引库 Elasticsearch之curl创建索引 Elasticsearch之curl创建索引库和索引时注意事项 Elasticsearch之cur ...

随机推荐

  1. 我要进大厂之大数据ZooKeeper知识点（1）

    01 让我们一起学大数据 老刘又回来啦!在实验室师兄师姐都找完工作之后,在结束各种科研工作之后,老刘现在也要为找工作而努力了,要开始大数据各个知识点的复习总结了.老刘会分享出自己的知识点总结,一是希望 ...

  2. Linux文件监控工具——inotify-tools

    举例: ip.txt内容如下: 10.1.1.11 root 123 10.1.1.22 root 111 10.1.1.33 root 123456 10.1.1.44 root 54321 写法1 ...

  3. kali 2020.1 更新源，并安装docker

    先说一句浙大牛逼!!!装个docker折腾了半天,测了半天只有浙大的更新源能用,完美不报错!清华阿里什么的更新源都是渣渣. deb http://mirrors.zju.edu.cn/kali kal ...

  4. python-验证码图片识别

    import tesserocr from PIL import Image #新建Image对象 image = Image.open('code.png') #调用tesserocr的image_ ...

  5. Cypress系列（90）- Cypress.Cookies 命令详解以及如何跨测试用例共享 Cookies

    如果想从头学起Cypress,可以看下面的系列文章哦 https://www.cnblogs.com/poloyy/category/1768839.html Cypress.Cookies 共有三个 ...

  6. 重新认识C++的"cin >>"、"cout <<" 简简单单 - 快快乐乐

    重新认识C++的"cin >>"."cout <<" 简简单单 - 快快乐乐 JERRY_Z. ~ 2020 / 11 / 24 转载请 ...

  7. 金九银十想去跳槽面试？那这份Java面经你真得看看了，写的非常详细！

    前言 前两天在和朋友吃饭的时候聊到时间这个东西是真的过的好坏啊,金三银四仿佛还在昨天.一眨眼金九银十又快到了,对程序员来说这两个是一年最合适的跳槽涨薪环节了,今年的你已经做好准备了吗?不妨看看这篇文章 ...

  8. 使用Camtasia来给视频或者图片调色

    喜欢摄影和制作视频的朋友可能知道,一张好看的照片或一段精美视频的构成要素很多,取景本身就是很重要的条件,相机的硬件水平也是一个重要因素,接下来的就是后期的编辑和处理了,而在后期处理过程中调色是十分重要 ...

  9. 浅析Nginx与Apache的区别

    Nginx相对于Apache的优势: 1.轻量级,采用C进行编写,同样的web服务,会占用更少的内存及资源 2.抗并发,nginx以epoll and kqueue作为开发模型,处理请求是异步非阻塞的 ...

  10. Java基础教程——内部类

    内部类 内部类(inner class)是定义在另一个类中的类 内部类的好处: |--1.隐藏机制:内部类封装性更好,隐藏在一个类之中,同一包中的其他类也不能访问 |--2.内部类可以访问外围类的私有 ...