Elastic Stack之Redis集群使用
Elastic Stack之Redis集群使用
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
本篇博客数据流走向：FileBeat ===》Redis ===》logstash ===> elasticsearch。
一.安装Redis服务器
1>.安装repo源
[root@node105 ~]#
[root@node105 ~]# yum -y install epel-release
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* extras: mirror.lzu.edu.cn
* updates: mirror.lzu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch :- will be installed
--> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================
Installing:
epel-release noarch - extras k Transaction Summary
===================================================================================================================================================================================================
Install Package Total download size: k
Installed size: k
Downloading packages:
epel-release--.noarch.rpm | kB ::
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
Installing : epel-release--.noarch /
Verifying : epel-release--.noarch / Installed:
epel-release.noarch :- Complete!
[root@node105 ~]#
[root@node105 ~]# yum -y install epel-release
2>.安装redis服务器
[root@node105 ~]#
[root@node105 ~]# yum -y install redis
Loaded plugins: fastestmirror
epel/x86_64/metalink | 7.9 kB ::
epel | 4.7 kB ::
(/): epel/x86_64/group_gz | kB ::
(/): epel/x86_64/updateinfo | kB ::
(/): epel/x86_64/primary_db | 6.6 MB ::
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirror.lzu.edu.cn
* updates: mirror.lzu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package redis.x86_64 :3.2.-.el7 will be installed
--> Processing Dependency: libjemalloc.so.()(64bit) for package: redis-3.2.-.el7.x86_64
--> Running transaction check
---> Package jemalloc.x86_64 :3.6.-.el7 will be installed
--> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================
Installing:
redis x86_64 3.2.-.el7 epel k
Installing for dependencies:
jemalloc x86_64 3.6.-.el7 epel k Transaction Summary
===================================================================================================================================================================================================
Install Package (+ Dependent package) Total download size: k
Installed size: 1.7 M
Downloading packages:
warning: /var/cache/yum/x86_64//epel/packages/jemalloc-3.6.-.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY ] 0.0 B/s | B --:--:-- ETA
Public key for jemalloc-3.6.-.el7.x86_64.rpm is not installed
(/): jemalloc-3.6.-.el7.x86_64.rpm | kB ::
(/): redis-3.2.-.el7.x86_64.rpm | kB ::
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total kB/s | kB ::
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Importing GPG key 0x352C64E5:
Userid : "Fedora EPEL (7) <epel@fedoraproject.org>"
Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
Package : epel-release--.noarch (@extras)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : jemalloc-3.6.-.el7.x86_64 /
Installing : redis-3.2.-.el7.x86_64 /
Verifying : redis-3.2.-.el7.x86_64 /
Verifying : jemalloc-3.6.-.el7.x86_64 / Installed:
redis.x86_64 :3.2.-.el7 Dependency Installed:
jemalloc.x86_64 :3.6.-.el7 Complete!
[root@node105 ~]#
[root@node105 ~]# yum -y install redis
3>.修改Redis服务器的默认配置
[root@node105 ~]#
[root@node105 ~]# grep bind /etc/redis.conf | grep -v ^# #指定绑定的ip地址为服务器的所有网卡！
bind 0.0.0.0
[root@node105 ~]#
[root@node105 ~]# grep requirepass /etc/redis.conf | grep -v ^# #开启认证
requirepass yinzhengjie.org.cn
[root@node105 ~]#
[root@node105 ~]#
4>.启动redis服务器
[root@node105 ~]#
[root@node105 ~]# systemctl enable redis
Created symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /usr/lib/systemd/system/redis.service.
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start redis
[root@node105 ~]#
[root@node105 ~]# systemctl status redis
● redis.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/redis.service.d
└─limit.conf
Active: active (running) since Mon -- :: CST; 4s ago
Main PID: (redis-server)
CGroup: /system.slice/redis.service
└─ /usr/bin/redis-server 0.0.0.0: Mar :: node105.yinzhengjie.org.cn systemd[]: Starting Redis persistent key-value database...
Mar :: node105.yinzhengjie.org.cn systemd[]: Started Redis persistent key-value database.
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ss -tnl | grep
LISTEN *: *:*
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start redis
二.使用FileBeat结合redis收集日志文件案例
1>.安装FileBeat软件
详情请参考： Elastic Stack之FileBeat使用实战。
2>.修改FileBeat的配置文件并启动
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"
filebeat.prospectors:
- input_type: log
paths:
- /var/log/httpd/access_log output.redis:
enabled: true
hosts: ["node105.yinzhengjie.org.cn:6379"]
key: filebeat
password: yinzhengjie.org.cn
db:
datatype: list
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #" #配置我们可以从/etc/filebeat/filebeat.full.yml将支持的输出配置拷贝过去！注意配置文件的属性之间应该有一个空格！
3>.查看redis服务器
[root@node105 ~]#
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
127.0.0.1:>
127.0.0.1:> KEYS *
) "filebeat"
127.0.0.1:>
127.0.0.1:> LINDEX filebeat
"{\"@timestamp\":\"2019-03-11T15:34:37.114Z\",\"beat\":{\"hostname\":\"node105.yinzhengjie.org.cn\",\"name\":\"node105.yinzhengjie.org.cn\",\"version\":\"5.6.12\"},\"input_type\":\"log\",\"message\":\"80.129.1.1 - - [11/Mar/2019:23:05:48 +0800] \\\"GET /test42.html HTTP/1.1\\\" 200 8 \\\"-\\\" \\\"curl/7.29.0\\\"\",\"offset\":134141,\"source\":\"/var/log/httpd/access_log\",\"type\":\"log\"}"
127.0.0.1:>
127.0.0.1:>
127.0.0.1:>
127.0.0.1:> LLEN filebeat
(integer)
127.0.0.1:> [root@node105 ~]# redis-cli -a yinzhengjie.org.cn
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
4>.配置logstash的redis输入插件（参考链接：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html）
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-filter-elasticsearch.conf
input {
redis {
host => "node105.yinzhengjie.org.cn"
port =>
password => "yinzhengjie.org.cn"
db =>
key => "filebeat"
data_type => "list"
}
} filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => ["message","beat"]
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
} output {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_access_logs"
}
} [root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-filter-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf -t
5>.启动logstash并查看redis服务器
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test59.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test53.html was not found on this server.</p>
</body></html>
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test55.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test58.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test60.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test52.html was not found on this server.</p>
</body></html>
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test54.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test57.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test51.html was not found on this server.</p>
</body></html>
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title> Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /test56.html was not found on this server.</p>
</body></html>
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
^C
[root@node103 ~]#
[root@node103 ~]# while true; do curl -H "X-Forwarded-For:$[$RANDOM%223+1].$[$RANDOM%255].1.1" http://node105.yinzhengjie.org.cn/test$[$RANDOM%60+1].html;sleep 1;done
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
127.0.0.1:>
127.0.0.1:> LLEN filebeat
(integer)
127.0.0.1:>
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn #启动logstash之前
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-filter-elasticsearch.conf #启动logstash
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn
127.0.0.1:>
127.0.0.1:>
127.0.0.1:> LLEN filebeat
(integer)
127.0.0.1:>
[root@node105 ~]# redis-cli -a yinzhengjie.org.cn #启动logstash之后
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
428k --:--:-- --:--:-- --:--:-- 441k
{
"took": ,
"timed_out": false,
"_shards": {
"total": ,
"successful": ,
"skipped": ,
"failed":
},
"hits": {
"total": ,
"max_score": 1.8790942,
"hits": [
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAEi",
"_score": 1.8790942,
"_source": {
"request": "/test55.html",
"geoip": {
"ip": "100.242.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:33.000Z",
"response": "",
"bytes": "",
"clientip": "100.242.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAEq",
"_score": 1.8790942,
"_source": {
"request": "/test52.html",
"geoip": {
"timezone": "Europe/Paris",
"ip": "82.234.1.1",
"latitude": 48.6942,
"continent_code": "EU",
"city_name": "Brunoy",
"country_name": "France",
"country_code2": "FR",
"country_code3": "FR",
"region_name": "Essonne",
"location": {
"lon": 2.4922,
"lat": 48.6942
},
"postal_code": "",
"region_code": "",
"longitude": 2.4922
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:41.000Z",
"response": "",
"bytes": "",
"clientip": "82.234.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAEt",
"_score": 1.8790942,
"_source": {
"request": "/test52.html",
"geoip": {
"timezone": "Africa/Accra",
"ip": "45.209.1.1",
"latitude": ,
"country_name": "Ghana",
"country_code2": "GH",
"continent_code": "AF",
"country_code3": "GH",
"location": {
"lon": -,
"lat":
},
"longitude": -
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:02:44.000Z",
"response": "",
"bytes": "",
"clientip": "45.209.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAE9",
"_score": 1.8790942,
"_source": {
"request": "/test58.html",
"geoip": {
"timezone": "Asia/Tokyo",
"ip": "111.89.1.1",
"latitude": 34.9833,
"continent_code": "AS",
"city_name": "Shobara",
"country_name": "Japan",
"country_code2": "JP",
"country_code3": "JP",
"region_name": "Hiroshima",
"location": {
"lon": 132.9833,
"lat": 34.9833
},
"postal_code": "727-0321",
"region_code": "",
"longitude": 132.9833
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:01.000Z",
"response": "",
"bytes": "",
"clientip": "111.89.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFH",
"_score": 1.8790942,
"_source": {
"request": "/test51.html",
"geoip": {
"timezone": "Europe/Madrid",
"ip": "83.47.1.1",
"latitude": 36.54,
"continent_code": "EU",
"city_name": "Fuengirola",
"country_name": "Spain",
"country_code2": "ES",
"country_code3": "ES",
"region_name": "Malaga",
"location": {
"lon": -4.6247,
"lat": 36.54
},
"postal_code": "",
"region_code": "MA",
"longitude": -4.6247
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:11.000Z",
"response": "",
"bytes": "",
"clientip": "83.47.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFY",
"_score": 1.8790942,
"_source": {
"request": "/test53.html",
"geoip": {
"timezone": "Europe/Rome",
"ip": "94.164.1.1",
"latitude": 42.5245,
"continent_code": "EU",
"city_name": "Piansano",
"country_name": "Italy",
"country_code2": "IT",
"country_code3": "IT",
"region_name": "Provincia di Viterbo",
"location": {
"lon": 11.8298,
"lat": 42.5245
},
"postal_code": "",
"region_code": "VT",
"longitude": 11.8298
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:28.000Z",
"response": "",
"bytes": "",
"clientip": "94.164.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFd",
"_score": 1.8790942,
"_source": {
"request": "/test55.html",
"geoip": {
"timezone": "Africa/Tunis",
"ip": "154.106.1.1",
"latitude": ,
"country_name": "Tunisia",
"country_code2": "TN",
"continent_code": "AF",
"country_code3": "TN",
"location": {
"lon": ,
"lat":
},
"longitude":
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:33.000Z",
"response": "",
"bytes": "",
"clientip": "154.106.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN80lXxXllWpXYAFq",
"_score": 1.8790942,
"_source": {
"request": "/test56.html",
"geoip": {
"ip": "33.65.1.1",
"latitude": 37.751,
"country_name": "United States",
"country_code2": "US",
"continent_code": "NA",
"country_code3": "US",
"location": {
"lon": -97.822,
"lat": 37.751
},
"longitude": -97.822
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:46.000Z",
"response": "",
"bytes": "",
"clientip": "33.65.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN85S3WCT5NaOiwHV",
"_score": 1.8790942,
"_source": {
"request": "/test57.html",
"geoip": {
"timezone": "America/Bogota",
"ip": "179.19.1.1",
"latitude": 4.5981,
"country_name": "Colombia",
"country_code2": "CO",
"continent_code": "SA",
"country_code3": "CO",
"location": {
"lon": -74.0758,
"lat": 4.5981
},
"longitude": -74.0758
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:03:54.000Z",
"response": "",
"bytes": "",
"clientip": "179.19.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
},
{
"_index": "logstash-2019.03.11",
"_type": "httpd_access_logs",
"_id": "AWltN85T3WCT5NaOiwHc",
"_score": 1.8790942,
"_source": {
"request": "/test51.html",
"geoip": {
"city_name": "Central",
"timezone": "Asia/Hong_Kong",
"ip": "13.94.1.1",
"latitude": 22.2909,
"country_name": "Hong Kong",
"country_code2": "HK",
"continent_code": "AS",
"country_code3": "HK",
"region_name": "Central and Western District",
"location": {
"lon": 114.15,
"lat": 22.2909
},
"region_code": "HCW",
"longitude": 114.15
},
"offset": ,
"auth": "-",
"ident": "-",
"input_type": "log",
"verb": "GET",
"source": "/var/log/httpd/access_log",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": "\"-\"",
"@timestamp": "2019-03-11T14:04:01.000Z",
"response": "",
"bytes": "",
"clientip": "13.94.1.1",
"@version": "",
"beat": {
"name": "node105.yinzhengjie.org.cn",
"hostname": "node105.yinzhengjie.org.cn",
"version": "5.6.12"
},
"host": "node105.yinzhengjie.org.cn",
"httpversion": "1.1",
"user_agent": "\"curl/7.29.0\""
}
}
]
}
}
[root@node101 ~]#
[root@node101 ~]# curl -X GET http://node101.yinzhengjie.org.cn:9200/logstash-*/_search?q=response:404 | jq . #查询数据
三.使用一个FileBeat加载多种不同的日志案例
1>.编写FileBeat的配置文件
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"
filebeat.prospectors:
- input_type: log
paths:
- /var/log/httpd/access_log
fields:
log_type: access
- paths:
- /var/log/httpd/error_log
fields:
log_type: errors
output.redis:
enabled: true
hosts: ["node105.yinzhengjie.org.cn:6379"]
key: filebeat
password: yinzhengjie.org.cn
db:
datatype: list
logging.level: debug
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/filebeat/filebeat.yml | egrep -v "^#|^$| #"
2>.编写logstash的配置文件
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-condition-elasticsearch.conf
input {
redis {
host => "node105.yinzhengjie.org.cn"
port =>
password => "yinzhengjie.org.cn"
db =>
key => "filebeat"
data_type => "list"
}
} filter {
if [fields][log_type] == "access" {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
remove_field => ["message","beat"]
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/maxmind/GeoLite2-City.mmdb"
}
}
} output {
if [fields][log_type] == "access" {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_access_logs"
}
} else {
elasticsearch {
hosts => ["http://node101.yinzhengjie.org.cn:9200/","http://node102.yinzhengjie.org.cn:9200/","http://node103.yinzhengjie.org.cn:9200/"]
index => "logstash-%{+YYYY.MM.dd}"
document_type => "httpd_error_logs"
}
}
} [root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# cat /etc/logstash/conf.d/redis-condition-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-condition-elasticsearch.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@node105 ~]#
[root@node105 ~]# logstash -f /etc/logstash/conf.d/redis-condition-elasticsearch.conf -t
3>.启动logstash服务
[root@node105 ~]#
[root@node105 ~]# mkdir -pv /yinzhengjie/logstash/conf.d
mkdir: created directory ‘/yinzhengjie/logstash’
mkdir: created directory ‘/yinzhengjie/logstash/conf.d’
[root@node105 ~]#
[root@node105 ~]# mv /etc/logstash/conf.d/* /yinzhengjie/logstash/conf.d/
[root@node105 ~]#
[root@node105 ~]# cp /yinzhengjie/logstash/conf.d/redis-condition-elasticsearch.conf /etc/logstash/conf.d/
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ll /etc/logstash/conf.d/
total 4
-rw-r--r--. 1 root root 1112 Mar 12 21:45 redis-condition-elasticsearch.conf
[root@node105 ~]#
[root@node105 ~]# cp /yinzhengjie/logstash/conf.d/redis-condition-elasticsearch.conf /etc/logstash/conf.d/ #我们将要用的配置文件存放到默认的目录中
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start logstash
[root@node105 ~]#
[root@node105 ~]# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
Active: active (running) since Tue -- :: CST; 5s ago
Main PID: (java)
CGroup: /system.slice/logstash.service
└─ /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction= -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-... Mar :: node105.yinzhengjie.org.cn systemd[]: Started logstash.
Mar :: node105.yinzhengjie.org.cn systemd[]: Starting logstash...
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# systemctl start logstash #启动logstash后，他会自动去默认的配置文件加载相应的配置文件！
[root@node105 ~]#
[root@node105 ~]# ps -ef | grep logstash | grep -v grep
logstash : ? :: /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction= -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF- -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# netstat -untalp | grep
tcp6 127.0.0.1: :::* LISTEN /java
tcp6 172.30.1.105: 172.30.1.101: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.102: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.103: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.105: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.103: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.102: ESTABLISHED /java
tcp6 172.30.1.105: 172.30.1.101: ESTABLISHED /java
[root@node105 ~]#
[root@node105 ~]#
[root@node105 ~]# ps -ef | grep logstash | grep -v grep #检查logstash的进程
Elastic Stack之Redis集群使用的更多相关文章
- redis集群配置
客户端分片 程序端实现 代理proxy,访问proxy,proxy指定redis保存位置. Twemproxy Redis cluster ,会造成一部分数据丢失,无中心化1.将数据自动切分(spli ...
- Docker Swarm redis 集群搭建
Docker Swarm redis 集群搭建 环境1: 系统:Linux Centos 7.4 x64 内核:Linux docker 3.10.0-693.2.2.el7.x86_64 Docke ...
- Redis集群生产环境源码安装
安装redis集群 根据各人单位生产环境用户搭建一.安装环境 操作系统:centos7.6 关闭防火墙.关闭selinux redis1:192.168.26.128 redis2:192.1 ...
- filebeat收集日志传输到Redis集群,logstash从Redis集群中拉取数据
前提:已配置好Redis集群,并设置的有统一的访问密码 架构是filebeat-->redis集群-->logstash->elasticsearch,需要修改filebeat的输出 ...
- logstash7.3版本不支持从redis集群中拉取数据
filebeat可以把收集到的日志传输到redis集群中,但是logstash如何从从redis集群中拉取数据的呢? ogstash使用的是7.3版本 经过查看官网文档,发现logstash7.3版本 ...
- 工具推荐-使用RedisInsight工具对Redis集群CURD操作及数据可视化和性能监控
关注「WeiyiGeek」公众号 设为「特别关注」每天带你玩转网络安全运维.应用开发.物联网IOT学习! 希望各位看友[关注.点赞.评论.收藏.投币],助力每一个梦想. 本章目录 目录 0x00 快速 ...
- Redis集群案例与场景分析
1.背景 Redis的出现确实大大地提高系统大并发能力支撑的可能性,转眼间Redis的最新版本已经是3.X版本了,但我们的系统依然继续跑着2.8,并很好地支撑着我们当前每天5亿访问量的应用系统.想当年 ...
- Java Spring mvc 操作 Redis 及 Redis 集群
本文原创,转载请注明:http://www.cnblogs.com/fengzheng/p/5941953.html 关于 Redis 集群搭建可以参考我的另一篇文章 Redis集群搭建与简单使用 R ...
- Redis集群搭建与简单使用
介绍安装环境与版本 用两台虚拟机模拟6个节点,一台机器3个节点,创建出3 master.3 salve 环境. redis 采用 redis-3.2.4 版本. 两台虚拟机都是 CentOS ,一台 ...
随机推荐
- CF821C Okabe and Boxes
题目链接 题目大意 模拟栈的操作,要求从1~n依次弹出,若不符合可以排序,且不会有不合法情况,求最少排序次数. 思路 由于一定没有不合法情况,所以每次不符合顺序时可以直接清空,遇到栈为空时仍然要求弹出 ...
- FFmpeg 将YUV数据转RGB
只要开始初始化一次,结束后释放就好,中间可以循环转码 AVFrame *m_pFrameRGB,*m_pFrameYUV; uint8_t *m_rgbBuffer,*m_yuvBuffer; str ...
- hdu 2955 Robberies (01背包）
链接:http://acm.hdu.edu.cn/showproblem.php?pid=2955 思路:一开始看急了,以为概率是直接相加的,wa了无数发,这道题目给的是被抓的概率,我们应该先求出总的 ...
- bzoj 4196 [Noi2015]软件包管理器 (树链剖分+线段树）
4196: [Noi2015]软件包管理器 Time Limit: 10 Sec Memory Limit: 512 MBSubmit: 2852 Solved: 1668[Submit][Sta ...
- IDEA调试技巧之条件断点
调试的时候,在循环里增加条件判断,可以极大的提高效率,心情也能愉悦.以下介绍下IDEA使用条件[Condition]断点的方法 1.编写一段样例代码 /** * @author jiashubing ...
- k短路模板（洛谷P2483 [SDOI2010]魔法猪学院）（k短路，最短路，左偏树，priority_queue）
你谷数据够强了,以前的A*应该差不多死掉了. 所以,小伙伴们快来一起把YL顶上去把!戳这里! 俞鼎力的课件 需要掌握的内容: Dijkstra构建最短路径树. 可持久化堆(使用左偏树,因其有二叉树结构 ...
- python3基础概念
数: str:使用单引号或双引号表达,不可变的,一旦创建不可更改,可以使变量赋予不同的字符串,但字符串本身是没有更改的: list:有序的集合,可变的基础数据类型: clear(),copy(),ap ...
- 一点理解之 CmBacktrace: ARM Cortex-M 系列 MCU 错误追踪库
@2019-02-14 [小记] CmBacktrace: ARM Cortex-M 系列 MCU 错误追踪库,用来将单片机故障状态寄存器值翻译出来输出至终端上以便排错 CmBacktrace: AR ...
- [WC2008]游览计划(状压dp)
题面太鬼畜不粘了. 题意就是给一张n*m的网格图,每个点有点权,有k个关键点,让你把这k个关键点连成一个联通快的最小代价. 题解 这题nmk都非常小,解法肯定是状压,比较一般的解法插头dp,但不太好写 ...
- 编写高质量代码:改善Java程序的151个建议 --[52~64]
编写高质量代码:改善Java程序的151个建议 --[52~64] 推荐使用String直接量赋值 Java为了避免在一个系统中大量产生String对象(为什么会大量产生,因为String字符串是程序 ...