Elastic Stack之Logstash进阶
Elastic Stack之Logstash进阶
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.使用GeoLite2和logstash 过滤插件的geoip案例
1>.GeoLite2概述
GeoLite2数据库是免费的IP地理定位数据库，与MaxMind的GeoIP2数据库相当，但不太准确。GeoLite2国家和城市数据库在每个月的第一个星期二更新。GeoLite2 ASN数据库每周二更新一次。官方网址：https://www.maxmind.com/en/home。
2>.下载GeoLite2的免费库（下载地址：https://dev.maxmind.com/geoip/geoip2/geolite2/）
[root@node105 ~]# ll
total
-rw-r--r--. root root Sep : logstash-5.6..rpm
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
---- ::-- https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
Resolving geolite.maxmind.com (geolite.maxmind.com)... 104.17.201.89, 104.17.200.89, ::::c959, ...
Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.17.201.89|:... connected.
HTTP request sent, awaiting response... OK
Length: (27M) [application/gzip]
Saving to: ‘GeoLite2-City.tar.gz’ %[===========================================================================================================================================================>] ,, 197KB/s in 1m 59s -- :: ( KB/s) - ‘GeoLite2-City.tar.gz’ saved [/] [root@node105 ~]#
[root@node105 ~]# ll
total
-rw-r--r--. root root Mar : GeoLite2-City.tar.gz
-rw-r--r--. root root Sep : logstash-5.6..rpm
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
3>.解压GeoLite并创建软连接
[root@node105 ~]#
[root@node105 ~]# mkdir /etc/logstash/maxmind
[root@node105 ~]#
[root@node105 ~]# ll
total
-rw-r--r--. root root Mar : GeoLite2-City.tar.gz
-rw-r--r--. root root Sep : logstash-5.6..rpm
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/
[root@node105 ~]#
[root@node105 ~]# ll /etc/logstash/maxmind/GeoLite2-City_20190305/
total
-rw-r--r--. Mar : COPYRIGHT.txt
-rw-r--r--. Mar : GeoLite2-City.mmdb
-rw-r--r--. Mar : LICENSE.txt
-rw-r--r--. Mar : README.txt
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# tar -xf GeoLite2-City.tar.gz -C /etc/logstash/maxmind/
[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/
‘/etc/logstash/maxmind/GeoLite2-City.mmdb’ -> ‘/etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb’
[root@node105 ~]#
[root@node105 ~]# ll /etc/logstash/maxmind/
total
drwxr-xr-x. Mar : GeoLite2-City_20190305
lrwxrwxrwx. root root Mar : GeoLite2-City.mmdb -> /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ln -sv /etc/logstash/maxmind/GeoLite2-City_20190305/GeoLite2-City.mmdb /etc/logstash/maxmind/
4>.编写logstash配置文件并测试语法（）
[root@node105 ~]#
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-stdout.conf /etc/logstash/conf.d/file-date-geoip-stdout.conf
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
} output {
stdout {
codec => rubydebug
}
} [root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-stdout.conf
5>.启动logstash的geoip相关配置文件（参考链接：https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-geoip.html）
[root@node103 ~]#
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
"request" => "/test35.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "Europe/London",
"ip" => "85.211.1.1",
"latitude" => 52.4768,
"continent_code" => "EU",
"city_name" => "Birmingham",
"country_name" => "United Kingdom",
"country_code2" => "GB",
"country_code3" => "GB",
"region_name" => "Birmingham",
"location" => {
"lon" => -1.9341,
"lat" => 52.4768
},
"postal_code" => "B16",
"region_code" => "BIR",
"longitude" => -1.9341
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "85.211.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test12.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "America/New_York",
"ip" => "108.5.1.1",
"latitude" => 40.7667,
"continent_code" => "NA",
"city_name" => "Union City",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "New Jersey",
"location" => {
"lon" => -74.0311,
"lat" => 40.7667
},
"postal_code" => "",
"region_code" => "NJ",
"longitude" => -74.0311
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "108.5.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test37.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "America/Chicago",
"ip" => "24.118.1.1",
"latitude" => 45.0139,
"continent_code" => "NA",
"city_name" => "Saint Paul",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "Minnesota",
"location" => {
"lon" => -93.1545,
"lat" => 45.0139
},
"postal_code" => "",
"region_code" => "MN",
"longitude" => -93.1545
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "24.118.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test38.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"ip" => "55.27.1.1",
"latitude" => 37.751,
"country_name" => "United States",
"country_code2" => "US",
"continent_code" => "NA",
"country_code3" => "US",
"location" => {
"lon" => -97.822,
"lat" => 37.751
},
"longitude" => -97.822
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "55.27.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test11.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"timezone" => "America/Los_Angeles",
"ip" => "3.173.1.1",
"latitude" => 47.6348,
"continent_code" => "NA",
"city_name" => "Seattle",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "Washington",
"location" => {
"lon" => -122.3451,
"lat" => 47.6348
},
"postal_code" => "",
"region_code" => "WA",
"longitude" => -122.3451
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "3.173.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
{
"request" => "/test14.html",
"agent" => "\"curl/7.29.0\"",
"geoip" => {
"city_name" => "Guayaquil",
"timezone" => "America/Guayaquil",
"ip" => "191.99.1.1",
"latitude" => -2.1664,
"country_name" => "Ecuador",
"country_code2" => "EC",
"continent_code" => "SA",
"country_code3" => "EC",
"region_name" => "Provincia del Guayas",
"location" => {
"lon" => -79.9011,
"lat" => -2.1664
},
"region_code" => "G",
"longitude" => -79.9011
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "191.99.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1"
}
^C[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-stdout.conf
二.logstash 过滤插件的Mutate案例
1>.mutate概述
mutate过滤器允许您在字段上执行常规突变。您可以重命名，删除，替换和修改事件中的字段。详情请参考：https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-mutate.html。
2>.编写mutate案例
[root@node105 ~]#
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
[root@node105 ~]#
[root@node105 ~]# vi /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
mutate {
rename => {
"agent" => "user_agent"
}
}
} output {
stdout {
codec => rubydebug
}
} [root@node105 ~]#
[root@node105 ~]# cp /etc/logstash/conf.d/file-date-geoip-stdout.conf /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf ^C
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
3>.启动案例
[root@node103 ~]#
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%50+1].html;sleep 1;done
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
{
"request" => "/test32.html",
"geoip" => {
"timezone" => "America/New_York",
"ip" => "73.137.1.1",
"latitude" => 33.9135,
"continent_code" => "NA",
"city_name" => "Powder Springs",
"country_name" => "United States",
"country_code2" => "US",
"dma_code" => ,
"country_code3" => "US",
"region_name" => "Georgia",
"location" => {
"lon" => -84.6859,
"lat" => 33.9135
},
"postal_code" => "",
"region_code" => "GA",
"longitude" => -84.6859
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "73.137.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1",
"user_agent" => "\"curl/7.29.0\""
}
{
"request" => "/test32.html",
"geoip" => {
"city_name" => "Daegu",
"timezone" => "Asia/Seoul",
"ip" => "119.201.1.1",
"latitude" => 35.8723,
"country_name" => "South Korea",
"country_code2" => "KR",
"continent_code" => "AS",
"country_code3" => "KR",
"region_name" => "Daegu",
"location" => {
"lon" => 128.5924,
"lat" => 35.8723
},
"region_code" => "",
"longitude" => 128.5924
},
"auth" => "-",
"ident" => "-",
"verb" => "GET",
"path" => "/var/log/httpd/access_log",
"referrer" => "\"-\"",
"@timestamp" => --11T13::.000Z,
"response" => "",
"bytes" => "",
"clientip" => "119.201.1.1",
"@version" => "",
"host" => "0.0.0.0",
"httpversion" => "1.1",
"user_agent" => "\"curl/7.29.0\""
}
^C[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-date-geoip-mutate-stdout.conf
三.logstash 输出插件之elasticsearch输出插件
1>.elasticsearch输出插件概述
此插件是在Elasticsearch中存储日志的推荐方法。如果您打算使用Kibana Web界面，则需要使用此输出。此输出仅说HTTP协议。从Logstash 2.0开始，HTTP是与Elasticsearch交互的首选协议。出于多种原因，我们强烈建议在节点协议上使用HTTP。HTTP只是稍微慢一点，但更容易管理和使用。使用HTTP协议时，可以升级Elasticsearch版本，而无需在锁定步骤中升级Logstash。官方文档：https://www.elastic.co/guide/en/logstash/5.6/plugins-outputs-elasticsearch.html。
2>.配置elasticsearch集群输出
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf
input {
file {
path => ["/var/log/httpd/access_log"]
start_position => "beginning"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
mutate {
rename => {
"agent" => "user_agent"
}
}
} output {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_access_logs"
}
} [root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/file-filter-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf -t
3>.运行logstash 配置文件并查看es集群是否有新的索引
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done #我改动了该脚本，运行时会访问不到某些网站，模拟404！
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[root@node105 ~]# logstash -f /etc/logstash/conf.d/file-filter-elasticsearch.conf #运行脚本，数据会被写入到es集群中
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
--:--:-- --:--:-- --:--:--
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": null,
"hits": []
}
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.2 | jq . #查询一条不存在的数据
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
--:--:-- --:--:-- --:--:--
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": 2.0794415,
"hits": [
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltCr5Hsru-A5a8RIhU",
"_score": 2.0794415,
"_source": {
"request": "/test17.html",
"geoip": {
"timezone": "America/Mexico_City",
"ip": "187.152.1.1",
"latitude": 20.6347,
"continent_code": "NA",
"city_name": "Guadalajara",
"country_name": "Mexico",
"country_code2": "MX",
"country_code3": "MX",
"region_name": "Jalisco",
"location": {
"lon": -103.4344,
"lat": 20.6347
},
"postal_code": "",
"region_code": "JAL",
"longitude": -103.4344
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T13:40:15.000Z",
"response": "",
"bytes": "",
"clientip": "187.152.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
}
]
}
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=clientip:187.152.1.1 | jq . #查询一条已经存在的数据
[root@node101 ~]#
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
250k --:--:-- --:--:-- --:--:-- 256k
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": 2.3795462,
"hits": [
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEH9tsru-A5a8RIhq",
"_score": 2.3795462,
"_source": {
"request": "/test51.html",
"geoip": {
"timezone": "Europe/Madrid",
"ip": "83.47.1.1",
"latitude": 36.54,
"continent_code": "EU",
"city_name": "Fuengirola",
"country_name": "Spain",
"country_code2": "ES",
"country_code3": "ES",
"region_name": "Malaga",
"location": {
"lon": -4.6247,
"lat": 36.54
},
"postal_code": "",
"region_code": "MA",
"longitude": -4.6247
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:11.000Z",
"response": "",
"bytes": "",
"clientip": "83.47.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEUMs3WCT5NaOiwE7",
"_score": 2.3795462,
"_source": {
"request": "/test51.html",
"geoip": {
"city_name": "Central",
"timezone": "Asia/Hong_Kong",
"ip": "13.94.1.1",
"latitude": 22.2909,
"country_name": "Hong Kong",
"country_code2": "HK",
"continent_code": "AS",
"country_code3": "HK",
"region_name": "Central and Western District",
"location": {
"lon": 114.15,
"lat": 22.2909
},
"region_code": "HCW",
"longitude": 114.15
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:04:01.000Z",
"response": "",
"bytes": "",
"clientip": "13.94.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltECF4sru-A5a8RIhi",
"_score": 2.0794415,
"_source": {
"request": "/test51.html",
"geoip": {
"timezone": "Europe/Oslo",
"ip": "78.91.1.1",
"latitude": 63.4167,
"continent_code": "EU",
"city_name": "Trondheim",
"country_name": "Norway",
"country_code2": "NO",
"country_code3": "NO",
"region_name": "Trøndelag",
"location": {
"lon": 10.4167,
"lat": 63.4167
},
"postal_code": "",
"region_code": "",
"longitude": 10.4167
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:46.000Z",
"response": "",
"bytes": "",
"clientip": "78.91.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD9sF3WCT5NaOiwEd",
"_score": 2.0794415,
"_source": {
"request": "/test57.html",
"geoip": {
"ip": "175.91.1.1",
"latitude": 34.7725,
"country_name": "China",
"country_code2": "CN",
"continent_code": "AS",
"country_code3": "CN",
"location": {
"lon": 113.7266,
"lat": 34.7725
},
"longitude": 113.7266
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:28.000Z",
"response": "",
"bytes": "",
"clientip": "175.91.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD-6fXxXllWpXYACG",
"_score": 2.0794415,
"_source": {
"request": "/test55.html",
"geoip": {
"ip": "100.242.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:33.000Z",
"response": "",
"bytes": "",
"clientip": "100.242.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD7u03WCT5NaOiwEZ",
"_score": 2.0794415,
"_source": {
"request": "/test59.html",
"geoip": {
"timezone": "Asia/Tokyo",
"ip": "126.210.1.1",
"latitude": 35.69,
"country_name": "Japan",
"country_code2": "JP",
"continent_code": "AS",
"country_code3": "JP",
"location": {
"lon": 139.69,
"lat": 35.69
},
"longitude": 139.69
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:20.000Z",
"response": "",
"bytes": "",
"clientip": "126.210.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEKqCsru-A5a8RIhw",
"_score": 2.0512707,
"_source": {
"request": "/test54.html",
"geoip": {
"timezone": "Asia/Tokyo",
"ip": "60.137.1.1",
"latitude": 34.9667,
"continent_code": "AS",
"city_name": "Nagoya",
"country_name": "Japan",
"country_code2": "JP",
"country_code3": "JP",
"region_name": "Aichi",
"location": {
"lon": 136.9667,
"lat": 34.9667
},
"postal_code": "470-2101",
"region_code": "",
"longitude": 136.9667
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:22.000Z",
"response": "",
"bytes": "",
"clientip": "60.137.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltD9Mu3WCT5NaOiwEc",
"_score": 2.0512707,
"_source": {
"request": "/test58.html",
"geoip": {
"ip": "12.254.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:26.000Z",
"response": "",
"bytes": "",
"clientip": "12.254.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltEVLT3WCT5NaOiwE9",
"_score": 2.0512707,
"_source": {
"request": "/test57.html",
"geoip": {
"timezone": "Asia/Shanghai",
"ip": "113.8.1.1",
"latitude": 45.75,
"country_name": "China",
"country_code2": "CN",
"continent_code": "AS",
"country_code3": "CN",
"region_name": "Heilongjiang",
"location": {
"lon": 126.65,
"lat": 45.75
},
"region_code": "HL",
"longitude": 126.65
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:04:04.000Z",
"response": "",
"bytes": "",
"clientip": "113.8.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltESfDsru-A5a8RIh5",
"_score": 2.0512707,
"_source": {
"request": "/test57.html",
"geoip": {
"timezone": "America/Bogota",
"ip": "179.19.1.1",
"latitude": 4.5981,
"country_name": "Colombia",
"country_code2": "CO",
"continent_code": "SA",
"country_code3": "CO",
"location": {
"lon": -74.0758,
"lat": 4.5981
},
"longitude": -74.0758
},
"auth": "-",
"ident": "-",
"verb": "GET",
"path": "/var/log/httpd/access_log",
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:54.000Z",
"response": "",
"bytes": "",
"clientip": "179.19.1.1",
"@version": "",
"host": "0.0.0.0",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
}
]
}
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq . #查询响应码为404的网站
Elastic Stack之Logstash进阶的更多相关文章
- 浅尝 Elastic Stack (二) Logstash
一.安装与启动 Logstash 依赖 Java 8 或者 Java 11,需要先安装 JDK 1.1 下载 curl -L -O https://artifacts.elastic.co/downl ...
- 浅尝 Elastic Stack (三) Logstash + Beats
本文使用 Filebeat,如果没有安装需要安装: curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat- ...
- 浅尝 Elastic Stack (五) Logstash + Beats + Kafka
在 Elasticsearch.Kibana.Beats 安装 中讲到推荐架构: 本文基于 Logstash + Beats 读取 Spring Boot 日志 将其改为上述架构 如果没有安装 Kaf ...
- 浅尝 Elastic Stack (四) Logstash + Beats 读取 Spring Boot 日志
一.Spring Boot 日志配置 采用 Spring Boot 默认的 Logback: <?xml version="1.0" encoding="UTF-8 ...
- Elastic Stack核心产品介绍-Elasticsearch、Logstash和Kibana
Elastic Stack 是一系列开源产品的合集,包括 Elasticsearch.Kibana.Logstash 以及 Beats 等等,能够安全可靠地获取任何来源.任何格式的数据,并且能够实时地 ...
- Elastic Stack（ElasticSearch 、 Kibana 和 Logstash） 实现日志的自动采集、搜索和分析
Elastic Stack 包括 Elasticsearch.Kibana.Beats 和 Logstash(也称为 ELK Stack).能够安全可靠地获取任何来源.任何格式的数据,然后实时地对数据 ...
- Elastic Stack
Elastic Stack 开发人员不能登陆线上服务器查看详细日志 各个系统都有日志,日志数据分散难以查找 日志数据量大,查询速度慢,或者数据不够实时 官网地址:https://www.elastic ...
- Elastic Stack之kibana入门
为了解决公司的项目在集群环境下查找日志不便的问题,我在做过简单调研后,选用Elastic公司的Elastic Stack产品作为我们的日志收集,存储,分析工具. Elastic Stack是ELK(E ...
- Elastic Stack之kibana使用
Elastic Stack之kibana使用 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客数据流走向:FileBeat ===>Redis ===>log ...
随机推荐
- java基础1之基本数据类型
java的数据类型 整数型(byte.short.int.long) 编程过程中,默认是int类型.long类型的字面值后面需要加上L或l PS:java底层,byte.short是按照32位计算的. ...
- 使用Java泛型返回动态类型
返回一个指定类型的集合,并且clazz必须继承IGeoLog对象或者是其本身 <T extends IGeoLog> List<T> getLogListSql(Class&l ...
- MySQL参数调优
目录 连接相关参数 文件相关参数 缓存相关参数 MyISAM参数 InnoDB参数 连接相关参数 max_connections 允许客户端并发连接的最大数量,默认值是151,一般将该参数设置为50 ...
- [POJ2976] Dropping tests
传送门:>Here< 题意:给出长度相等的数组a和b,定义他们的和为$\dfrac{a_1+a_2+...+a_n}{b_1+b_2+...+b_n}$.现在可以舍弃k对元素(一对即$a[ ...
- 慢腾腾的Quartus prime16.0加快编译速度
前言 当一个工程反复修改的时候,可能有时候源代码没有更改,为了加快编译速度可以配置quartus一些选项.当然,初次编译的速度是否会提升,未验证.更高级的设计分区以及逻辑锁区提升速度,以后阐述. 流程 ...
- PHP获取网络图片并保存在本地目录
PHP获取网络图片并保存在本地目录思路: 代码如下: function file_exists_S3($url) { $state = @file_get_contents($url,0,null,0 ...
- MT【250】距离0-7
是否存在一个正方体,它的8个顶点到某一个平面的距离恰好为$0,1,2,3,4,5,6,7$ ?若存在指出正方体与相应的平面的位置关系.不存在说明理由. 分析:设平面$\alpha$的单位法向量为$\o ...
- 【UOJ#340】【清华集训2017】小 Y 和恐怖的奴隶主（矩阵快速幂，动态规划）
[UOJ#340][清华集训2017]小 Y 和恐怖的奴隶主(矩阵快速幂,动态规划) 题面 UOJ 洛谷 题解 考虑如何暴力\(dp\). 设\(f[i][a][b][c]\)表示当前到了第\(i\) ...
- IncDec Sequence（差分）
题意:给定一个序列,可以对一个区间进行加1或减1的操作,问最少需要多少次可以将序列的值一样. Solution 我们将序列差分,得到一个差分数组. 对于每一个区间操作,我们可以把它转化为在查分数组上某 ...
- selenium js
这几天的任务量比较大,还有一个挺棘手的网站cfda,不巧的是数据量还挺大,40W关于企业信息.上来就是debugger pause,调试中断,开始还是挺懵逼的,但这个还算简单毕竟google,百度,就 ...