Configuring Logstash

To configure Logstash, you create a config file that specifies which plugins you want to use and settings for each plugin.

You can reference event fields in a configuration and use conditionals to process events when they meet certain criteria.

When you run logstash, you use the -f to specify your config file.

Let’s step through creating a simple config file and using it to run Logstash. Create a file named "logstash-simple.conf" and save it in the same directory as Logstash.

input { stdin { } }
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

Then, run logstash and specify the configuration file with the -f flag.

bin/logstash -f logstash-simple.conf

Et voilà! Logstash reads the specified configuration file and outputs to both Elasticsearch and stdout.

Before we move on to some more complex examples, let’s take a closer look at what’s in a config file.

Structure of a Config File

A Logstash config file has a separate section for each type of plugin you want to add to the event processing pipeline.

For example:

# This is a comment. You should use comments to describe
# parts of your configuration.
input {
...
} filter {
...
} output {
...
}

Each section contains the configuration options for one or more plugins.

If you specify multiple filters, they are applied in the order of their appearance in the configuration file.

Plugin Configuration

The configuration of a plugin consists of the plugin name followed by a block of settings for that plugin.

For example, this input section configures two file inputs:

input {
file {
path => "/var/log/messages"
type => "syslog"
} file {
path => "/var/log/apache/access.log"
type => "apache"
}
}

In this example, two settings are configured for each of the file inputs: path and type.

The settings you can configure vary according to the plugin type.

For information about each plugin, see Input Plugins, Output Plugins, Filter Plugins, and Codec Plugins.

Value Types

A plugin can require that the value for a setting be a certain type, such as boolean, list, or hash.

The following value types are supported.

Array

This type is now mostly deprecated弃用 in favor of using a standard type like string with the plugin defining the :list => true property for better type checking.

It is still needed to handle lists of hashes or mixed types where type checking is not desired.

Example:

  users => [ {id => 1, name => bob}, {id => 2, name => jane} ]

Lists

Not a type in and of itself, but a property types can have.

This makes it possible to type check multiple values.

Plugin authors can enable list checking by specifying :list => true when declaring an argument.

Example:

path => [ "/var/log/messages", "/var/log/*.log" ]
uris => [ "http://elastic.co", "http://example.net" ]

This example configures path, which is a string to be a list that contains an element for each of the three strings.

It also will configure the uris parameter to be a list of URIs, failing if any of the URIs provided are not valid.

Boolean

A boolean must be either true or false.

Note that the true and false keywords are not enclosed in quotes.

Example:

ssl_enable => true

Bytes

A bytes field is a string field that represents a valid unit of bytes.

It is a convenient way to declare specific sizes in your plugin options.

Both SI (k M G T P E Z Y) and Binary (Ki Mi Gi Ti Pi Ei Zi Yi) units are supported.

Binary units are in base-1024 and SI units are in base-1000.

This field is case-insensitive and accepts space between the value and the unit.

If no unit is specified, the integer string represents the number of bytes.

Examples:

  my_bytes => "1113"   # 1113 bytes
my_bytes => "10MiB" # 10485760 bytes
my_bytes => "100kib" # 102400 bytes
my_bytes => "180 mb" # 180000000 bytes

Codec

A codec is the name of Logstash codec used to represent the data. Codecs can be used in both inputs and outputs.

Input codecs provide a convenient way to decode your data before it enters the input.

Output codecs provide a convenient way to encode your data before it leaves the output.

Using an input or output codec eliminates the need for a separate filter in your Logstash pipeline.

A list of available codecs can be found at the Codec Plugins page.

Example:

  codec => "json"

Hash

A hash is a collection of key value pairs specified in the format "field1" => "value1".

Note that multiple key value entries are separated by spaces rather than commas.

Example:

match => {
"field1" => "value1"
"field2" => "value2"
...
}

Number

Numbers must be valid numeric values (floating point or integer).

Example:

port => 33

Password

A password is a string with a single value that is not logged or printed.

Example:

my_password => "password"

URI

A URI can be anything from a full URL like http://elastic.co/ to a simple identifier like foobar.

If the URI contains a password such as http://user:pass@example.net the password portion of the URI will not be logged or printed.

Example:

my_uri => "http://foo:bar@example.net"

Path

A path is a string that represents a valid operating system path.

Example:

String

A string must be a single character sequence. Note that string values are enclosed in quotes, either double or single.

Escape Sequences

By default, escape sequences are not enabled.

If you wish to use escape sequences in quoted strings, you will need to set config.support_escapes: true in your logstash.yml.

When true, quoted strings (double and single) will have this transformation:

name => "Hello world"
name => 'It\'s a beautiful day'

Comments

Comments are the same as in perl, ruby, and python. A comment starts with a # character, and does not need to be at the beginning of a line. For example:

# this is a comment

input { # comments can appear at the end of a line, too
# ...
}

Configuring Logstash的更多相关文章

  1. 【原创】大数据基础之Logstash(4)高可用

    logstash高可用体现为不丢数据(前提为服务器短时间内不可用后可恢复比如重启服务器或重启进程),具体有两个方面: 进程重启(服务器重启) 事件消息处理失败 在logstash中对应的解决方案为: ...

  2. 2-4、配置Filebeat使用logstash

    配置filebeat使用logstash 重要:要将事件发送到Logstash,还需要创建一个Logstash配置管道,该管道监听传入的Beats连接并将收到的事件编入索引到Elasticsearch ...

  3. Kubernetes部署ELK并使用Filebeat收集容器日志

    本文的试验环境为CentOS 7.3,Kubernetes集群为1.11.2,安装步骤参见kubeadm安装kubernetes V1.11.1 集群 1. 环境准备 Elasticsearch运行时 ...

  4. Windows上怎么安装ELK

    In this guide I will show that it is also possible to run Logstash on a Windows Server 2012 machine ...

  5. 【笔记】草履虫也能看懂的ELK搭建流程

    环境需要 Elasticsearch需要JAVA环境,至少是JDK1.8 elasticsearch 不允许root用户使用,需要新增个elk用户 如果觉得官网下载太慢,可以使用这个 https:// ...

  6. 测试右移:线上质量监控 ELK 实战

    目录 [测试右移]介绍 ELK Stack 介绍 ELK 监控体系搭建 ES & Kibana 搭建 Nginx 日志自动采集 Nginx Agent 安装 Nginx 服务器 数据分析 Lo ...

  7. Manage Spring Boot Logs with Elasticsearch, Logstash and Kibana

    下载地址:https://www.elastic.co/downloads When time comes to deploy a new project, one often overlooked ...

  8. ElasticSearch+Logstash+Filebeat+Kibana集群日志管理分析平台搭建

    一.ELK搜索引擎原理介绍 在使用搜索引擎是你可能会觉得很简单方便,只需要在搜索栏输入想要的关键字就能显示出想要的结果.但在这简单的操作背后是搜索引擎复杂的逻辑和许多组件协同工作的结果. 搜索引擎的组 ...

  9. elasticsearch+logstash+redis+kibana 实时分析nginx日志

    1. 部署环境 2. 架构拓扑 3. nginx安装 安装在192.168.176.128服务器上 这里安装就简单粗暴了直接yum安装nginx [root@manager ~]# yum -y in ...

随机推荐

  1. Python之words count

    要求: 对文件单词进行统计,不区分大小写,并显示单词重复最多的十个单词 思路: 利用字典key,value的特性存单词及其重复的次数 每行进行特殊字符的处理,分离出被特殊字符包含的单词 def mak ...

  2. 前端-CSS样式

    一.CSS介绍 CSS(Cascading Style Sheet),全称层叠样式,定义如何显示HTML内的元素,浏览器读取HTML文件时,读取到CSS样式时根据CSS规则来对内容进行渲染 1.CSS ...

  3. 仿豆瓣网(电脑版网页)HTML+CSS实现

    步骤一:将豆瓣电脑版网页以图片形式保存下来: 利用了chrome里面的插件: 步骤二:将图片放置到PS中,研究布局: 我将其分为header部分,banner部分,section部分,footer部分 ...

  4. workerman程序调试

    现象1 启动后报错类似如下: php start.php start PHP Warning: stream_socket_server(): unable to connect to tcp://x ...

  5. html5常用数学 公式的用法

    <script>        // alert(Math.PI);        // alert(Math.floor(3.16));    // var a=Math.ceil(3. ...

  6. tomcat的JVM调优

    1.error场景 Tomcat 长期运行过程遇到Caused by: java.lang.OutOfMemoryError: PermGen space或java.lang.OutOfMemoryE ...

  7. Spark学习之路 (八)SparkCore的调优之开发调优

    摘抄自:https://tech.meituan.com/spark-tuning-basic.html 前言 在大数据计算领域,Spark已经成为了越来越流行.越来越受欢迎的计算平台之一.Spark ...

  8. python 将字节字符串转换成十六进制字符串

    想将一个十六进制字符串解码成一个字节字符串或者将一个字节字符串编码成一个十六进制字符串 1)只是简单的解码或编码一个十六进制的原始字符串,可以使用 binascii模块 >>> s ...

  9. android搜索框列表布局,流程及主要步骤思维导图

    android搜索框列表布局,流程及主要步骤思维导图 android搜索框列表布局,流程及主要步骤思维导图 activity_coin_search.xml----------<com.scwa ...

  10. window下nodejs用nodemon启动koa2项目(用cmd启动不了,要用Git Bash Here 启动才可以)

    window下nodejs用nodemon启动koa2项目(用cmd启动不了,要用Git Bash Here 启动才可以)nodemon --watch 'app/**/*' -e ts --exec ...