Nginx访问日志分析

nginx默认的日志格式

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

字段说明

127.0.0.1 - - [14/May/2017:12:51:13 +0800] "GET /index.html HTTP/1.1" 200 4286 "http://127.0.0.1/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36"
远程主机IP            请求时间       时区  方法    资源      协议   状态码 发送字节    referer      浏览器信息

统计访问IP前十

# awk '{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
   6958 123.174.51.164
   2307 111.85.34.165
   1617 118.112.143.148
   1489 117.63.146.40
   1404 118.182.116.39
   1352 1.48.219.30
   1132 60.222.231.46
   1129 10.35.1.82
    943 27.227.163.200
    880 58.253.6.133

统计指定某一天的访问IP

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10
# awk '/17\/May\/2017/{print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
   6958 123.174.51.164
   2307 111.85.34.165
   1617 118.112.143.148
   1489 117.63.146.40
   1404 118.182.116.39
   1352 1.48.219.30
   1132 60.222.231.46
   1129 10.35.1.82
    943 27.227.163.200
    880 58.253.6.133

经过测试，在文件较大的时候，先grep再awk速度快很多。

过滤URL

# awk '{print $11}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
  20737 "http://www.adreambox.net/index.php?app=home&mod=User&act=index"
   4155 "-"
   3981 "http://www.adreambox.net/"
   1921 "http://www.adreambox.net/index.php?app=adreambox&mod=Class&act=prensent&id=5&type=2"
   1299 "http://www.adreambox.net/index.php?app=home&mod=Public&act=doLogin"
   1191 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1413"
    718 "http://www.adreambox.net/index.php?app=group&mod=Group&act=index&gid=1403"
    657 "http://www.adreambox.net/index.php?app=wap&mod=Index&act=index"
    657 "http://www.adreambox.net/index.php?act=index&app=home&mod=User"
    639 "http://www.adreambox.net/index.php?app=group&mod=Manage&act=index&gid=1413"

统计指定资源

# awk '($7~/\.html$/){print $1 " " $7 " " $9}' /usr/local/nginx/logs/access.log     #处理第7个字段以'.html'结尾的行
11.0.8.5 //ckeditor/notexist_path.html 404
11.0.8.5 //ckeditor/CHANGES.html 404
11.0.8.18 //docs/CHANGELOG.html 404
11.0.8.5 //themes/mall/default/seller_order.confirm.html 404
11.0.8.18 //themes/mall/default/header.html 404
11.0.8.5 //themes/store/default/footer.html 404
11.0.8.5 //templates/admin/index.html 404
11.0.8.5 //system/templates/admin/login.html 404
11.0.8.18 //templates/404.html 404
11.0.8.18 //admin/editor/editor/dialog/fck_about.html 404
11.0.8.5 //fckeditor/_whatsnew.html 404
11.0.8.5 //FCKeditor/_docs/whatsnew.html 404
11.0.8.5 //style/gb/help/index.html 404
10.10.1.11 /Login/login.html 404

过滤指定时间后的日志并打印IP

# awk '($4>"[15/May/2017:21:16:38"){print $1}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr
 291031 11.0.8.5
 274174 11.0.8.18
   2764 10.10.1.11
   1193 11.0.8.6
      1 127.0.0.1

统计流量

# grep "17/May/2017" /usr/local/nginx/logs/access.log | awk '{sum+=$10}END{print sum}'
95210093059

统计状态码

# awk '{print $9}' /usr/local/nginx/logs/access.log | sort | uniq -c | sort -nr | head -10
1271257 200
 957444 503
  61875 502
  32852 404
  19121 302
  13356 304
   2819 500
   2789 400
    271 499
    203 401

过滤某个时间段的日志

# sed -n '/2017-5-18 9:51:13/,/2017-5-18 9:55:13/p' access.log