Elastic Stack之Logstash进阶

                                        作者:尹正杰 

版权声明:原创作品,谢绝转载!否则将追究法律责任。

一.使用GeoLite2和logstash 过滤插件的geoip案例

1>.GeoLite2概述

  GeoLite2数据库是免费的IP地理定位数据库,与MaxMind的GeoIP2数据库相当,但不太准确。GeoLite2国家和城市数据库在每个月的第一个星期二更新。GeoLite2 ASN数据库每周二更新一次。官方网址:https://www.maxmind.com/en/home

2>.下载GeoLite2的免费库(下载地址:https://dev.maxmind.com/geoip/geoip2/geolite2/

[root@node105 ~]# ll
total
-rw-r--r--. root root Sep : logstash-5.6..rpm
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
---- ::-- https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
Resolving geolite.maxmind.com (geolite.maxmind.com)... 104.17.201.89, 104.17.200.89, ::::c959, ...
Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.17.201.89|:... connected.
HTTP request sent, awaiting response... OK
Length: (27M) [application/gzip]
Saving to: ‘GeoLite2-City.tar.gz’ %[===========================================================================================================================================================>] ,, 197KB/s in 1m 59s -- :: ( KB/s) - ‘GeoLite2-City.tar.gz’ saved [/] [root@node105 ~]#
[root@node105 ~]# ll
total
-rw-r--r--. root root Mar : GeoLite2-City.tar.gz
-rw-r--r--. root root Sep : logstash-5.6..rpm
[root@node105 ~]#
[root@node105 ~]#

[root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz

3>.解压GeoLite并创建软连接

[root@node105 ~]#
[root@node105 ~]# mkdir /etc/logstash/maxmind
[root@node105 ~]#
[root@node105 ~]# ll
total
-rw-r--r--. root root Mar : GeoLite2-City.tar.gz
-rw-r--r--. root root Sep : logstash-5.6..rpm
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/
[root@node105 ~]#
[root@node105 ~]# ll /etc/logstash/maxmind/GeoLite2-City_20190305/
total
-rw-r--r--. Mar : COPYRIGHT.txt
-rw-r--r--. Mar : GeoLite2-City.mmdb
-rw-r--r--. Mar : LICENSE.txt
-rw-r--r--. Mar : README.txt
[root@node105 ~]#
[root@node105 ~]#

[root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/

[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/
‘/etc/logstash/maxmind/GeoLite2-City.mmdb’ -> ‘/etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb’
[root@node105 ~]#
[root@node105 ~]# ll /etc/logstash/maxmind/
total
drwxr-xr-x. Mar : GeoLite2-City_20190305
lrwxrwxrwx. root root Mar : GeoLite2-City.mmdb -> /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb
[root@node105 ~]#
[root@node105 ~]#

[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/

4>.编写logstash配置文件并测试语法()

[root@node105 ~]#
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-stdout.conf /etc/logstash/conf.d/file-date-geoip-stdout.conf
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
} output {
stdout {
codec => rubydebug
}
} [root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#

[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf

5>.启动logstash的geoip相关配置文件(参考链接:https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-geoip.html

[root@node103 ~]#
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
^C
[root@node103 ~]#

[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done

[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
"request" => "/test35.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "Europe/London",
"ip" => "85.211.1.1",
"latitude" => 52.4768,
"continent_code" => "EU",
"city_name" => "Birmingham",
"country_name" => "United Kingdom",
"country_code2" => "GB",
"country_code3" => "GB",
"region_name" => "Birmingham",
"location" => {
"lon" => -1.9341,
"lat" => 52.4768
},
"postal_code" => "B16",
"region_code" => "BIR",
"longitude" => -1.9341
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "85.211.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test12.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "America/New_York",
"ip" => "108.5.1.1",
"latitude" => 40.7667,
"continent_code" => "NA",
"city_name" => "Union City",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "New Jersey",
"location" => {
"lon" => -74.0311,
"lat" => 40.7667
},
"postal_code" => "",
"region_code" => "NJ",
"longitude" => -74.0311
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "108.5.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test37.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "America/Chicago",
"ip" => "24.118.1.1",
"latitude" => 45.0139,
"continent_code" => "NA",
"city_name" => "Saint Paul",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "Minnesota",
"location" => {
"lon" => -93.1545,
"lat" => 45.0139
},
"postal_code" => "",
"region_code" => "MN",
"longitude" => -93.1545
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "24.118.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test38.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"ip" => "55.27.1.1",
"latitude" => 37.751,
"country_name" => "United States",
"country_code2" => "US",
"continent_code" => "NA",
"country_code3" => "US",
"location" => {
"lon" => -97.822,
"lat" => 37.751
},
"longitude" => -97.822
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "55.27.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test11.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "America/Los_Angeles",
"ip" => "3.173.1.1",
"latitude" => 47.6348,
"continent_code" => "NA",
"city_name" => "Seattle",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "Washington",
"location" => {
"lon" => -122.3451,
"lat" => 47.6348
},
"postal_code" => "",
"region_code" => "WA",
"longitude" => -122.3451
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "3.173.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test14.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"city_name" => "Guayaquil",
"timezone" => "America/Guayaquil",
"ip" => "191.99.1.1",
"latitude" => -2.1664,
"country_name" => "Ecuador",
"country_code2" => "EC",
"continent_code" => "SA",
"country_code3" => "EC",
"region_name" => "Provincia del Guayas",
"location" => {
"lon" => -79.9011,
"lat" => -2.1664
},
"region_code" => "G",
"longitude" => -79.9011
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "191.99.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
^C[root@node105 ~]#

[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf

二.logstash 过滤插件的Mutate案例

1>.mutate概述

  mutate过滤器允许您在字段上执行常规突变。您可以重命名,删除,替换和修改事件中的字段。详情请参考:https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-mutate.html

2>.编写mutate案例

[root@node105 ~]#
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
[root@node105 ~]#
[root@node105 ~]# vi /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
mutate {
rename => {
"agent" => "user_agent"
}
}
} output {
stdout {
codec => rubydebug
}
} [root@node105 ~]#
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf ^C
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#

[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf

3>.启动案例

[root@node103 ~]#
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
^C
[root@node103 ~]#

[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done

[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
"request" => "/test32.html",
"geoip" => {
"timezone" => "America/New_York",
"ip" => "73.137.1.1",
"latitude" => 33.9135,
"continent_code" => "NA",
"city_name" => "Powder Springs",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "Georgia",
"location" => {
"lon" => -84.6859,
"lat" => 33.9135
},
"postal_code" => "",
"region_code" => "GA",
"longitude" => -84.6859
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "73.137.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1",
"user_agent" => "\"curl/7.29.0\""
}
{
"request" => "/test32.html",
"geoip" => {
"city_name" => "Daegu",
"timezone" => "Asia/Seoul",
"ip" => "119.201.1.1",
"latitude" => 35.8723,
"country_name" => "South Korea",
"country_code2" => "KR",
"continent_code" => "AS",
"country_code3" => "KR",
"region_name" => "Daegu",
"location" => {
"lon" => 128.5924,
"lat" => 35.8723
},
"region_code" => "",
"longitude" => 128.5924
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "119.201.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1",
"user_agent" => "\"curl/7.29.0\""
}
^C[root@node105 ~]#

[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf

三.logstash 输出插件之elasticsearch输出插件

1>.elasticsearch输出插件概述

  此插件是在Elasticsearch中存储日志的推荐方法。如果您打算使用Kibana Web界面,则需要使用此输出。此输出仅说HTTP协议。从Logstash 2.0开始,HTTP是与Elasticsearch交互的首选协议。出于多种原因,我们强烈建议在节点协议上使用HTTP。HTTP只是稍微慢一点,但更容易管理和使用。使用HTTP协议时,可以升级Elasticsearch版本,而无需在锁定步骤中升级Logstash。官方文档:https://www.elastic.co/guide/en/logstash/5.6/plugins-outputs-elasticsearch.html

2>.配置elasticsearch集群输出

[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
mutate {
rename => {
"agent" => "user_agent"
}
}
} output {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_access_logs"
}
} [root@node105 ~]#

[root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf

[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#

[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t

3>.运行logstash 配置文件并查看es集群是否有新的索引

[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
^C
[root@node103 ~]#

[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done    #我改动了该脚本,运行时会访问不到某些网站,模拟404!

[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf                                    #运行脚本,数据会被写入到es集群中

[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
--:--:-- --:--:-- --:--:--
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": null,
"hits": []
}
}
[root@node101 ~]#

[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq .              #查询一条不存在的数据

[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
--:--:-- --:--:-- --:--:--
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": 2.0794415,
"hits": [
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltCr5Hsru-A5a8RIhU",
"_score": 2.0794415,
"_source": {
"request": "/test17.html",
"geoip": {
"timezone": "America/Mexico_City",
"ip": "187.152.1.1",
"latitude": 20.6347,
"continent_code": "NA",
"city_name": "Guadalajara",
"country_name": "Mexico",
"country_code2": "MX",
"country_code3": "MX",
"region_name": "Jalisco",
"location": {
"lon": -103.4344,
"lat": 20.6347
},
"postal_code": "",
"region_code": "JAL",
"longitude": -103.4344
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T13:40:15.000Z",
"response": "",
"bytes": "",
"clientip": "187.152.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
}
]
}
}
[root@node101 ~]#

[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq .              #查询一条已经存在的数据

[root@node101 ~]#
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
250k --:--:-- --:--:-- --:--:-- 256k
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": 2.3795462,
"hits": [
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEH9tsru-A5a8RIhq",
"_score": 2.3795462,
"_source": {
"request": "/test51.html",
"geoip": {
"timezone": "Europe/Madrid",
"ip": "83.47.1.1",
"latitude": 36.54,
"continent_code": "EU",
"city_name": "Fuengirola",
"country_name": "Spain",
"country_code2": "ES",
"country_code3": "ES",
"region_name": "Malaga",
"location": {
"lon": -4.6247,
"lat": 36.54
},
"postal_code": "",
"region_code": "MA",
"longitude": -4.6247
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:11.000Z",
"response": "",
"bytes": "",
"clientip": "83.47.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEUMs3WCT5NaOiwE7",
"_score": 2.3795462,
"_source": {
"request": "/test51.html",
"geoip": {
"city_name": "Central",
"timezone": "Asia/Hong_Kong",
"ip": "13.94.1.1",
"latitude": 22.2909,
"country_name": "Hong Kong",
"country_code2": "HK",
"continent_code": "AS",
"country_code3": "HK",
"region_name": "Central and Western District",
"location": {
"lon": 114.15,
"lat": 22.2909
},
"region_code": "HCW",
"longitude": 114.15
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:04:01.000Z",
"response": "",
"bytes": "",
"clientip": "13.94.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltECF4sru-A5a8RIhi",
"_score": 2.0794415,
"_source": {
"request": "/test51.html",
"geoip": {
"timezone": "Europe/Oslo",
"ip": "78.91.1.1",
"latitude": 63.4167,
"continent_code": "EU",
"city_name": "Trondheim",
"country_name": "Norway",
"country_code2": "NO",
"country_code3": "NO",
"region_name": "Trøndelag",
"location": {
"lon": 10.4167,
"lat": 63.4167
},
"postal_code": "",
"region_code": "",
"longitude": 10.4167
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:46.000Z",
"response": "",
"bytes": "",
"clientip": "78.91.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD9sF3WCT5NaOiwEd",
"_score": 2.0794415,
"_source": {
"request": "/test57.html",
"geoip": {
"ip": "175.91.1.1",
"latitude": 34.7725,
"country_name": "China",
"country_code2": "CN",
"continent_code": "AS",
"country_code3": "CN",
"location": {
"lon": 113.7266,
"lat": 34.7725
},
"longitude": 113.7266
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:28.000Z",
"response": "",
"bytes": "",
"clientip": "175.91.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD-6fXxXllWpXYACG",
"_score": 2.0794415,
"_source": {
"request": "/test55.html",
"geoip": {
"ip": "100.242.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:33.000Z",
"response": "",
"bytes": "",
"clientip": "100.242.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD7u03WCT5NaOiwEZ",
"_score": 2.0794415,
"_source": {
"request": "/test59.html",
"geoip": {
"timezone": "Asia/Tokyo",
"ip": "126.210.1.1",
"latitude": 35.69,
"country_name": "Japan",
"country_code2": "JP",
"continent_code": "AS",
"country_code3": "JP",
"location": {
"lon": 139.69,
"lat": 35.69
},
"longitude": 139.69
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:20.000Z",
"response": "",
"bytes": "",
"clientip": "126.210.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEKqCsru-A5a8RIhw",
"_score": 2.0512707,
"_source": {
"request": "/test54.html",
"geoip": {
"timezone": "Asia/Tokyo",
"ip": "60.137.1.1",
"latitude": 34.9667,
"continent_code": "AS",
"city_name": "Nagoya",
"country_name": "Japan",
"country_code2": "JP",
"country_code3": "JP",
"region_name": "Aichi",
"location": {
"lon": 136.9667,
"lat": 34.9667
},
"postal_code": "470-2101",
"region_code": "",
"longitude": 136.9667
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:22.000Z",
"response": "",
"bytes": "",
"clientip": "60.137.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD9Mu3WCT5NaOiwEc",
"_score": 2.0512707,
"_source": {
"request": "/test58.html",
"geoip": {
"ip": "12.254.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:26.000Z",
"response": "",
"bytes": "",
"clientip": "12.254.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEVLT3WCT5NaOiwE9",
"_score": 2.0512707,
"_source": {
"request": "/test57.html",
"geoip": {
"timezone": "Asia/Shanghai",
"ip": "113.8.1.1",
"latitude": 45.75,
"country_name": "China",
"country_code2": "CN",
"continent_code": "AS",
"country_code3": "CN",
"region_name": "Heilongjiang",
"location": {
"lon": 126.65,
"lat": 45.75
},
"region_code": "HL",
"longitude": 126.65
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:04:04.000Z",
"response": "",
"bytes": "",
"clientip": "113.8.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltESfDsru-A5a8RIh5",
"_score": 2.0512707,
"_source": {
"request": "/test57.html",
"geoip": {
"timezone": "America/Bogota",
"ip": "179.19.1.1",
"latitude": 4.5981,
"country_name": "Colombia",
"country_code2": "CO",
"continent_code": "SA",
"country_code3": "CO",
"location": {
"lon": -74.0758,
"lat": 4.5981
},
"longitude": -74.0758
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:54.000Z",
"response": "",
"bytes": "",
"clientip": "179.19.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
}
]
}
}
[root@node101 ~]#

[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .                    #查询响应码为404的网站

 

Elastic Stack之Logstash进阶的更多相关文章

  1. 浅尝 Elastic Stack (二) Logstash

    一.安装与启动 Logstash 依赖 Java 8 或者 Java 11,需要先安装 JDK 1.1 下载 curl -L -O https://artifacts.elastic.co/downl ...

  2. 浅尝 Elastic Stack (三) Logstash + Beats

    本文使用 Filebeat,如果没有安装需要安装: curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat- ...

  3. 浅尝 Elastic Stack (五) Logstash + Beats + Kafka

    在 Elasticsearch.Kibana.Beats 安装 中讲到推荐架构: 本文基于 Logstash + Beats 读取 Spring Boot 日志 将其改为上述架构 如果没有安装 Kaf ...

  4. 浅尝 Elastic Stack (四) Logstash + Beats 读取 Spring Boot 日志

    一.Spring Boot 日志配置 采用 Spring Boot 默认的 Logback: <?xml version="1.0" encoding="UTF-8 ...

  5. Elastic Stack核心产品介绍-Elasticsearch、Logstash和Kibana

    Elastic Stack 是一系列开源产品的合集,包括 Elasticsearch.Kibana.Logstash 以及 Beats 等等,能够安全可靠地获取任何来源.任何格式的数据,并且能够实时地 ...

  6. Elastic Stack(ElasticSearch 、 Kibana 和 Logstash) 实现日志的自动采集、搜索和分析

    Elastic Stack 包括 Elasticsearch.Kibana.Beats 和 Logstash(也称为 ELK Stack).能够安全可靠地获取任何来源.任何格式的数据,然后实时地对数据 ...

  7. Elastic Stack

    Elastic Stack 开发人员不能登陆线上服务器查看详细日志 各个系统都有日志,日志数据分散难以查找 日志数据量大,查询速度慢,或者数据不够实时 官网地址:https://www.elastic ...

  8. Elastic Stack之kibana入门

    为了解决公司的项目在集群环境下查找日志不便的问题,我在做过简单调研后,选用Elastic公司的Elastic Stack产品作为我们的日志收集,存储,分析工具. Elastic Stack是ELK(E ...

  9. Elastic Stack之kibana使用

    Elastic Stack之kibana使用 作者:尹正杰  版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客数据流走向:FileBeat ===>Redis  ===>log ...

随机推荐

  1. mpvue——引入antv-F2图表

    踩坑中~ 官方文档 https://www.yuque.com/antv/f2/intro 毕竟不像echarts接触过,所以还是先看看文档较好 github https://github.com/s ...

  2. POJ 1012

    参考自:https://www.cnblogs.com/ECJTUACM-873284962/p/6480880.html Joseph Time Limit: 1000MS   Memory Lim ...

  3. 【XSY2032】简单粗暴的题目 组合数

    题目描述 给你\(n,k,a_1\ldots a_n\),设 \[ ans_n=\sum_{i=1}^n{(\sum_{j=i}^ns(j))}^k\\ \] 求\(ans_1\ldots ans_n ...

  4. shopNC 拓扑图

    shopNC :

  5. DP及其优化

    常见DP模型及其构造 序列DP ARC074 RGB Sequence 题意 给你一个长度为 \(n\) 的序列和 \(m\) 组约束条件,每组条件形如 \(l_i,r_i,x_i\),表示序列上的 ...

  6. JLOI2015 DAY1 简要题解

    「JLOI2015」有意义的字符串 题意 给你 \(b, d, n\) 求 \[ [(\frac{b + \sqrt d}2)^n] \mod 7528443412579576937 \] \(0 & ...

  7. python3 列表list

    列表用中括号表示[]: list()创建一个列表: 是可变的: 可以被迭代,也可以被切片: +组合列表,*重复列表: 可以使用del删除元素,del L[index]; 方法: append(obj) ...

  8. Nginx代理MysqlCluster集群(二)

    Nginx代理MySql集群本次实验采用nginx 版本1.12以上 集合了tcp代理功能只需在编译时明文开启指定的功能 --with-stream--prefix=/usr/local/ngin - ...

  9. SQL查询语句的进阶使用

    MySQL的进阶使用 sql语句一些功能的使用 导入现有大量数据文件步骤 1) 把*.sql文件拷贝到Linux某一位置(例如Desktop) 2) Linux命令行进入该位置 cd ~/Deskto ...

  10. A1131. Subway Map (30)

    In the big cities, the subway systems always look so complex to the visitors. To give you some sense ...