502 Bad Gateway深究

早上收到502报警，设置的报警规则是502错误两分钟超过500就报警。

排障流程:

　　日志分析系统报障-->查看日志系统日志-->nginx错误日志-->php错误日志-->php-fpm.log日志

在日志分析系统里面看到产生502报警的机器只有一台xxx.xxx.xxx.170,客户端IP也只有一个，说明不是大规模故障。

连接到170服务器查看nginx错误日志，看到Connection reset by peer，连接被对方重置，说明php关掉了这个连接

// :: [error] #: * recv() failed (: Connection reset by peer) while reading response header from upstream, client: 123.206.95.156, server: jifen..com, request: "GET /center/index HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "jifen.51.com", referrer: "http://jifen.51.com/"
// :: [error] #: * recv() failed (: Connection reset by peer) while reading response header from upstream, client: 123.206.95.156, server: jifen..com, request: "GET /center/index HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "jifen.51.com", referrer: "http://jifen.51.com/"
// :: [error] #: * recv() failed (: Connection reset by peer) while reading response header from upstream, client: 123.206.95.156, server: jifen..com, request: "GET /center/index HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "jifen.51.com", referrer: "http://jifen.51.com/"
// :: [error] #: * recv() failed (: Connection reset by peer) while reading response header from upstream, client: 123.206.95.156, server: jifen..com, request: "GET /center/index HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "jifen.51.com", referrer: "http://jifen.51.com/"

继续查看日志，看php的错误日志，里面什么也没有。

然后查看php-fpm.log日志文件，看到index.php脚本执行超时，都是30多秒。

[-Dec- ::] WARNING: [pool www] child  exited on signal  (SIGTERM) after 10765.182746 seconds from start
[-Dec- ::] NOTICE: [pool www] child  started
[-Dec- ::] WARNING: [pool www] child  exited on signal  (SIGTERM) after 10287.255584 seconds from start
[-Dec- ::] NOTICE: [pool www] child  started
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (31.152266 sec), terminating
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (31.370696 sec), terminating
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (30.236601 sec), terminating
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (31.273963 sec), terminating
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (30.752849 sec), terminating
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (31.055392 sec), terminating
[-Dec- ::] WARNING: [pool www] child , script '/opt/wwwroot/jifen.51.com/www/index.php' (request: "GET /center/index") execution timed out (30.037908 sec), terminating

打开php-fpm.conf配置文件，里面设置的request_terminate_timeout = 30 设置的单一脚本执行超过30s就会被终止。

并且php.ini配置文件里max_execution_time = 60，设置单一脚本最多执行60s。

错误原因找到了，产生502 Bad Gateway的原因是这个脚本执行超时。鉴于是单一情况，只有一个用户出现502错误，不是所有用户连接都出现502，也为了安全及减轻服务器压力，未调大request_terminate_timeout值。