Nginx 服务器性能Bug和性能优化方案（真实经历）

一、遇到的问题

1、问题：本应该是3个ffmpeg ，但是怎么会有5个ffmpeg出现？

2、Lua脚本问题，一直写入日志，导致有大量的日志，这里的错误日志是直接写进nginx的error.log 日志文件的

（1）日志文件

total .8G
-rw-r--r-- 1 root root  11M May  9 21:47 access.log
-rw-r--r-- 1 root root 3.1K May  9 21:46 error.log
-rw-r--r-- 1 root root 705M May  9 21:44 live_access.log
-rw-r--r-- 1 root root 1.1G May  9 21:44 nginx_error.log

2017/05/09 21:43:05 时间段详细记录：

// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" matches "/live/C
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "
// :: [notice] #: * "\.m3u8" does not match "

2017/05/09 21:36:43 时间段详细记录，太恐怖了，1s写入日志多大240条，难怪nginx的负载飙到54%

// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [info] #: * client pre
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [info] #: * client
// :: [info] #: * client
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"
// :: [notice] #: * "\.m3u8"
// :: [error] #: * [lua] hls
// :: [notice] #: * "\.m3u8"

(2)Lua脚本日志写入：

if ( tonumber(res) ==  or tonumber(res) ==  ) then          -- 不是签名验证，则直接返回
   ngx.log(ngx.ERR, "error : 不是签名验证，则直接返回::res=== ",res)
    return
end

解决办法：立即关闭该日志的写入，重启nginx服务器 reload

【1】日志切割工具下载：http://soft.laozuo.org/tools/Knife4File.zip

【2】log日志查看工具：http://www.uvviewsoft.com/logviewer/files/LogViewerPro.zip

3、他们摄像头推流，直播结束后ffmpeg 进程一直在

4、是否针对播流权限认真是否取消掉？

答：暂时先取消掉

6、流状态列表查看（必须的联系他们的人员怎么处理），没有推流但是nginx服务状态一直存在

 解决方案：

修改nginx rtmp配置参数，添加这个参数：drop_idle_publisher 10s;

drop_idle_publisher
语法：drop_idle_publisher timeout
上下文：rtmp, server, application
描述：终止指定时间内闲置(没有音频/视频数据)的发布连接。默认为 off。注意这个仅仅对于发布模式的连接起作用(发送 publish 命令之后)。

 

二、性能优化技巧

1、设置worker_connections：10240

官方解释如下，个人认为是每一个worker进程能并发处理（发起）的最大连接数（包含所有连接数）。不能超过最大文件打开数：在linux终端中输入ulimit -a进行查看

www@ubuntu5:$ ulimit -a
core file size          (blocks, -c)
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e)
file size               (blocks, -f) unlimited
pending signals                 (-i)
max locked memory       (kbytes, -l)
max memory size         (kbytes, -m) unlimited
open files                      (-n)
pipe size            ( bytes, -p)
POSIX message queues     (bytes, -q)
real-time priority              (-r)
stack size              (kbytes, -s)
cpu time               (seconds, -t) unlimited
max user processes              (-u) 126277
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

参考：linux ulimit命令调优系统参数

 2、编译安装过程优化

 

找到源码的这个文件 /nginx-1.10.3/auto/cc/gcc

# debug
CFLAGS="$CFLAGS -g"

不修改代码大小7.3M：

tinywan@tinywan:~/nginx-1.10./auto/cc$ ls -lh /usr/local/nginx/sbin/nginx
-rwxr-xr-x  root root 7.3M 3月   : /usr/local/nginx/sbin/nginx

修改后的文件大小759K

tinywan@tinywan:~/nginx-1.10.$ ls -lh /usr/local/nginx/sbin/nginx
-rwxr-xr-x  root root 759K 5月   : /usr/local/nginx/sbin/nginx

查看CPU类型

tinywan@tinywan:~/nginx-1.10.$ cat /proc/cpuinfo | grep "model name"
model name    : Intel(R) Core(TM) i3- CPU @ .60GHz

 

 

 Nginx日志分割及自动删除超过30天日志

#!/bin/bash
# 分割nginx access日志脚本

time=`date +%Y%m%d`
logs_path="/usr/local/nginx/logs/"
pid_path="/usr/local/nginx/logs/nginx.pid"

#日志分割，按天创建文件夹
cd $logs_path
website=`ls $logs_path/access_* | xargs -n  | cut -f  -d "_"`
for i in $website
do
  mkdir -p $logs_path/backup/$time/
  mv $logs_path/access_$i $logs_path/backup/$time/$i
done
#向nginx主进程发信号重新打开日志
kill -USR1 `cat ${pid_path}`

#每月1日删除所有超过30天日志。
if [ "`date +%d`" = "" ]; then
  all_list=`ls $logs_path/backup | xargs -n `
  for del in $all_list
  do
    let results=$time-$del
    if [ $results -gt  ]; then
      rm -rf $logs_path/backup/$del
    fi
  done
fi

注： /usr/local/nginx/logs/下的日志格式为access_domain1.log。

加入计划任务，每天大概0时分割日志：

  * * * /data/script/nginx_logs.sh