Nginx 高级配置-自定义json格式日志

　　　　　　　　　　　　　　Nginx高级配置-自定义json格式日志

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　在大数据运维工作中，我们经常会使用flume，filebeat相关日志收集工具取收集日志，但这些日志在收集前都日志基本上都是json格式的，通过flume收集日志到hdfs集群，开发人员就直接使用java，scala语言取处理日志，有的时候会使用到spark,fink等框架去处理日志。因此nginx配置为json格式还是非常有必要的。

　　访问日志是记录客户端即用户的具体请求内容信息，全局配置模块中的error_log是记录nginx服务器运行时的日志保存路径和记录日志的level，因此有着本质的区别，而且Nginx的错误日志一般只有一个，但是访问日志可以在不同server中定义多个，定义一个日志需要使用access_log指定日志的保存路径，使用log_format指定日志的格式，格式中定义要保存的具体日志内容。

一.自定义默认格式日志

如果是要保留日志的源格式，只是添加相应的日志内容，则配置如下。

关于nginx日志使用的变量名称含义，博主推荐阅读:
　　https://www.cnblogs.com/yinzhengjie/p/12046613.html

1>.编写主配置文件

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/nginx/conf/nginx.conf
worker_processes  4;
worker_cpu_affinity 00000001 00000010 00000100 00001000; 

events {
    worker_connections  100000;
    use epoll;
    accept_mutex on;
    multi_accept on;
}

http {
    include       mime.types;

    default_type  text/html;

    charset utf-8;

    log_format my_default_format '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent "$http_referer"' '"$http_user_agent"' '"$http_x_forwarded_
for"' '$server_name:$server_port';
    access_log logs/access.log my_default_format;

    include /yinzhengjie/softwares/nginx/conf.d/*.conf;
}

[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[root@node101.yinzhengjie.org.cn ~]# 

2>.编写子配置文件

[root@node101.yinzhengjie.org.cn ~]# cat /yinzhengjie/softwares/nginx/conf.d/share.conf
server {
    listen 80;
    server_name node101.yinzhengjie.org.cn;

    location / {
        root /yinzhengjie/data/web/nginx/static;
        index index.html;
    }

    location /nginx_status {
        stub_status;
        allow 172.30.1.108;
    deny all;
    }

    location /main {
    index index.html;
    default_type text/html;
        set $name jason;
        set $nginx_name $server_name;
        echo "姓名: $name";
        echo "************";
        echo "Nginx服务器名称: $nginx_name";
    }

}
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[root@node101.yinzhengjie.org.cn ~]# 

3>.重新加载nginx的配置文件

[root@node101.yinzhengjie.org.cn ~]# ps -ef | grep nginx | grep -v grep
root      9297     1  0 Dec17 ?        00:00:00 nginx: master process nginx
nginx    11823  9297  0 12:50 ?        00:00:00 nginx: worker process
nginx    11824  9297  0 12:50 ?        00:00:00 nginx: worker process
nginx    11825  9297  0 12:50 ?        00:00:00 nginx: worker process
nginx    11826  9297  0 12:50 ?        00:00:00 nginx: worker process
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# nginx -s reload
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# ps -ef | grep nginx | grep -v grep
root      9297     1  0 Dec17 ?        00:00:00 nginx: master process nginx
nginx    11890  9297  1 12:57 ?        00:00:00 nginx: worker process
nginx    11891  9297  1 12:57 ?        00:00:00 nginx: worker process
nginx    11892  9297  1 12:57 ?        00:00:00 nginx: worker process
nginx    11893  9297  1 12:57 ?        00:00:00 nginx: worker process
[root@node101.yinzhengjie.org.cn ~]# 

4>.浏览器访问"http://node101.yinzhengjie.org.cn/main"并查看日志格式,如下图所示。

二.自定义json格式日志

　　Nginx的默认访问日志记录内容相对比较单一，默认的格式也不方便后期做日志统计分析，生产环境中通常将nginx日志转换为json日志，然后配合使用ELK做日志收集-统计-分析。

1>.编辑主配置文件

[root@node101.yinzhengjie.org.cn ~]# cat -n /yinzhengjie/softwares/nginx/conf/nginx.conf
     1    worker_processes  4;
     2    worker_cpu_affinity 00000001 00000010 00000100 00001000;
     3
     4    events {
     5        worker_connections  100000;
     6        use epoll;
     7        accept_mutex on;
     8        multi_accept on;
     9    }
    10
    11    http {
    12        include       mime.types;
    13
    14        default_type  text/html;
    15
    16        charset utf-8;
    17
    18        log_format my_access_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"uri":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"tcp_xff":"$proxy_protocol_addr",' '"http_user_agent":"$http_user_agent",' '"status":"$status"}';    
　　 19
    20
    21        access_log logs/access_json.log my_access_json;
    22
    23        include /yinzhengjie/softwares/nginx/conf.d/*.conf;
    24    }
    25
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[root@node101.yinzhengjie.org.cn ~]# 

2>.编辑子配置文件

[root@node101.yinzhengjie.org.cn ~]# cat -n /yinzhengjie/softwares/nginx/conf.d/share.conf
     1    server {
     2        listen 80;
     3        server_name node101.yinzhengjie.org.cn;
     4
     5        location / {
     6            root /yinzhengjie/data/web/nginx/static;
     7            index index.html;
     8        }
     9
    10        location /nginx_status {
    11            stub_status;
    12            allow 172.30.1.108;
    13        deny all;
    14        }
    15
    16        location /main {
    17        index index.html;
    18        default_type text/html;
    19            set $name jason;
    20            set $nginx_name $server_name;
    21            echo "姓名: $name";
    22            echo "************";
    23            echo "Nginx服务器名称: $nginx_name";
    24        }
    25
    26    }
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# 

3>.重新加载nginx配置文件

[root@node101.yinzhengjie.org.cn ~]# ps -ef | grep nginx | grep -v grep
root      9297     1  0 Dec17 ?        00:00:00 nginx: master process nginx
nginx    11890  9297  0 12:57 ?        00:00:00 nginx: worker process
nginx    11891  9297  0 12:57 ?        00:00:00 nginx: worker process
nginx    11892  9297  0 12:57 ?        00:00:00 nginx: worker process
nginx    11893  9297  0 12:57 ?        00:00:00 nginx: worker process
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# nginx -s reload
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# ps -ef | grep nginx | grep -v grep
root      9297     1  0 Dec17 ?        00:00:00 nginx: master process nginx
nginx    11946  9297  1 13:25 ?        00:00:00 nginx: worker process
nginx    11947  9297  1 13:25 ?        00:00:00 nginx: worker process
nginx    11948  9297  0 13:25 ?        00:00:00 nginx: worker process
nginx    11949  9297  1 13:25 ?        00:00:00 nginx: worker process
[root@node101.yinzhengjie.org.cn ~]# 

4>.浏览器访问"http://node101.yinzhengjie.org.cn/main"并查看日志格式,如下图所示。