记一次网络原因导致的mysql连接中断问题(druid)

date: 2018-04-19 21:00
tag: java,mysql,exception,mat,调试,jvm
工具: gceasy.io, MAT

线上系统出现一个诡异的bug，通过heap dump分析

分析:

通过日志确认系统在一天前就已经停止运行
代码较简单应该不存在DB写入操作死锁
mysql操作不是特别频繁

定位:

使用jmap -dump导出线上的应用的heap dump
jstack 导出堆栈信息
分析jstack发现多个Thread在DB写入时在等待同一把锁
通过MAT分析dump
通过OQL在MAT中定位上面出现的锁住多个Thread的锁对象
通过分析发现锁对象的所有者是阿里druid线程池的com.alibaba.druid.pool.DruidDataSource
在IDE中查看DruidDataSource的源码发现几个关键属性: poolingCount, errorCount, destroyCount, lastError
google搜索lastError的描述Communications link failure, 得到可能是网络原因导致的mysql连接关闭
在MAT中查看poolingCount, errorCount, destroyCount属性的值发现与上述结论一致
重启应用，问题消失

思考:

为什么Druid线程池没有尝试重新连接?
需要优化日志，提高排查效率
增加必要的Metric和预警规则

程序假死时的heap dump:

OQL:

 SELECT s.poolingCount,

 s.errorCount.toString() AS errorCount,

 s.destroyCount.toString() AS destroyCount,

 s.createCount.toString() AS createCount,

 s.lastError.exceptionMessage.toString()

 FROM com.alibaba.druid.pool.DruidDataSource s

结果:

 s.poolingCount |errorCount|destroyCount|createCount|s.lastError.exceptionMessage.toString()

---------------------------------------------------------------------------------------------------------------------------------------------

              0 |10        |7           |17         |Communications link failure\u000a\u000aLast packet sent to the server was 180289 ms ago.

---------------------------------------------------------------------------------------------------------------------------------------------

s.connections:

结果:

Type|Name|Value

---------------

ref |[0] |null

ref |[1] |null

ref |[2] |null

ref |[3] |null

ref |[4] |null

ref |[5] |null

ref |[6] |null

ref |[7] |null

---------------

正常运行时候的core dump:

OQL:

 SELECT s.poolingCount,

 s.errorCount.toString() AS errorCount,

 s.destroyCount.toString() AS destroyCount,

 s.createCount.toString() AS createCount,

 s.lastError.exceptionMessage.toString()

 FROM com.alibaba.druid.pool.DruidDataSource s

结果:

 s.poolingCount |errorCount|destroyCount|createCount|s.lastError.exceptionMessage.toString()

--------------------------------------------------------------------------------------------

              1 |1         |0           |4          |

--------------------------------------------------------------------------------------------

s.connections:

结果:

Type|Name|Value

-------------------------------------------------------------------

ref |[0] |com.alibaba.druid.pool.DruidConnectionHolder @ 0xc5fb1958

ref |[1] |null

ref |[2] |null

ref |[3] |null

ref |[4] |null

ref |[5] |null

ref |[6] |null

ref |[7] |null

-------------------------------------------------------------------

日志文件中的异常堆栈:

org.springframework.dao.RecoverableDataAccessException:

### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

### The error may exist in class path resource [mapper/ConfigMapper.xml]

### The error may involve com.myserver1.crawler.dao.ConfigDao.selectAllByType-Inline

### The error occurred while setting parameters

### SQL: select           id, type, props, content, status, create_time, update_time         from config     where     type = ?     and status = 1

### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

; SQL []; Communications link failure

Last packet sent to the server was 655505 ms ago.; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

        at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98) ~[spring-jdbc-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:74) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:399) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        at com.sun.proxy.$Proxy60.selectList(Unknown Source) ~[?:?]

        at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:205) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        at com.myserver1.crawler.dao.ConfigDao.selectAllByType(ConfigDao.java:27) ~[classes!/:1.0-SNAPSHOT]

        at com.myserver1.crawler.dao.ConfigDao$$FastClassBySpringCGLIB$$98d41dde.invoke(<generated>) ~[classes!/:1.0-SNAPSHOT]

        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.3.6.RELEASE.jar!/:4.3.6.RELEAS

E]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at com.myserver1.crawler.dao.ConfigDao$$EnhancerBySpringCGLIB$$c890af99.selectAllByType(<generated>) ~[classes!/:1.0-SNAPSHOT]

        at com.myserver1.event.VideoItemListener.post(VideoItemListener.java:32) ~[classes!/:1.0-SNAPSHOT]

        at com.myserver1.event.EventBus.push(EventBus.java:28) [classes!/:1.0-SNAPSHOT]

        at com.myserver1.crawler.site.ABreadthCrawler.start(ABreadthCrawler.java:97) [classes!/:1.0-SNAPSHOT]

        at com.myserver1.event.QueueThread.run(QueueThread.java:41) [classes!/:1.0-SNAPSHOT]

Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_101]

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_101]

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_101]

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_101]

        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3246) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:477) ~[druid-0.2.9.jar!/:0.2.9]

        at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:78) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113) ~[mybatis-3.3.0.jar!/:3.3.0]

        at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_101]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]

        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:386) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        ... 15 more

Caused by: java.net.SocketException: Connection reset

        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[?:1.8.0_101]

        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_101]

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_101]

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_101]

        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:477) ~[druid-0.2.9.jar!/:0.2.9]

        at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:78) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113) ~[mybatis-3.3.0.jar!/:3.3.0]

        at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_101]

记一次网络原因导致的mysql连接中断问题(druid)的更多相关文章

网络原因导致 npm 软件包 node-sass / gulp-sass 安装失败的处理办法
如果你正在构建一个基于 gulp 的前端自动化开发环境,那么极有可能会用到 gulp-sass ,由于网络原因你可能会安装失败,因为安装过程中部分细节会到亚马逊云服务器上获取文件.本文主要讨论在不变更 ...
网络原因导致的 spring cloud config 读取git上的配置文件时报错：Cannot clone or checkout repository
今天在公司使用spring cloud config搭建配置中心的时候,出现了读取不到git库的问题:Cannot clone or checkout repository.在网上百度,前面几个答案都 ...
记一次低级错误导致的mysql（111）
今天下午配好的双主多从服务器,两台主机+主机内安装好的6台虚拟机,两台Mysql master各授权好其slave的远程登录,原本好端端的能远程登录,晚上回来时候就发现其中一台master登录不上其s ...
Java Mysql连接池配置和案例分析--超时异常和处理
前言: 最近在开发服务的时候, 发现服务只要一段时间不用, 下次首次访问总是失败. 该问题影响虽不大, 但终究影响用户体验. 观察日志后发现, mysql连接因长时间空闲而被关闭, 使用时没有死链检测 ...
MongoDB3.4安装配置以及与Robomongo1.1的连接——解决Authentication Failed导致的不能连接问题
本文环境:win10(64)+MongoDB(3.4.5)+Robomongo(1.1) 目录: MongoDB的安装 MongoDB的配置 Robomongo的安装以及与MongoDB的连接一些新 ...
关于流量升高导致TIME_WAIT增加，MySQL连接大量失败的问题
有个应用就是每次都会去查一个接口,接口返回用户的信息数据,从而展现不同的页面效果.大致流程如下应用APP(电信)-> memcache ->电信custom接口 ->master- ...
SQLServer 2012之AlwaysOn —— 指定数据同步链路，消除网络抖动导致的提交延迟问题
事件起因:近期有研发反应,某数据库从08切换到12环境后,不定期出现写操作提交延迟的问题: 事件分析:在排除了系统资源争用等问题后,初步分析可能由于网络抖动导致同步模式alwayson节点经常出现会话 ...
记录一次mongodb因网络问题导致shard节点异常
现象: 机房反馈9点左右,机房交换机故障,导致网络出现问题业务人员反馈某个接口超时初查:通过业务日志查看分析发现,在连接mongo的某个collections时候,报错错误如下: 在写入数据的时候 ...
electron-builder 由于网络原因无法下载问题解决
electron-builder 由于网络原因无法下载问题解决在package.json的build中添加electron的镜像 "electronDownload": { &q ...

随机推荐

Python小札
在Python中,等号(=)是赋值语句,可以把任意数据类型赋值给变量,同一个变量可以反复赋值,而且可以是不同类型的变量如: a = 123 #a是整数 print(a) a = 'ABC' #a变为字 ...
api接口开发跨域注意事项和设置
因为跨域请求会先发送一个OPTIONS请求,所以需要判断下OPTIONS请求的就直接返回 if(strtoupper($_SERVER['REQUEST_METHOD'])== 'OPTIONS'){ ...
【算法】单源最短路——Dijkstra
对于固定起点的最短路算法,我们称之为单源最短路算法.单源最短路算法很多,最常见的就是dijkstra算法. dijkstra主要用的是一种贪心的思想,就是说如果i...s...t...j是最短路,那么 ...
python dict to dataframe
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_dict.html Examples By de ...
tensorflow tfrecord文件存储
import tensorflow as tf import numpy as np import skimage from skimage import data, io, color from P ...
left join on 后and 和 where 的区别
SELECT * FROM student a LEFT JOIN sc b ON a.Sid = b.Sid AND a.Sname="赵雷" 结果:(left join 左连接 ...
C++ MySQL编程
MySQL编程需要包含<mysql.h>头文件.该文件一般在MySQL安装目录下的include文件夹下. 包含头文件还不够,还需要包含“libmysql.lib”库,一般在lib文件夹下 ...
jquery添加属性使用attr、prop。
之前页面为标签添加属性都是使用的attr,删除使用removeAttr. 今天给checkbox添加checked属性时出现代码显示添加成功,但是页面不勾选内容. 后来查询发现checked是chec ...
微信小程序拒绝授权，反复调起原生授权框。
最近堕落了,有一阵子没有更新博客园了.一是比较忙,其次也没什么好的题材和工作中的解决方案可以分享的,想想还是把罕见的反复调起原生小程序授权框的方案拿出来说说. 市面上常见的解决方案是第一次拒绝后, ...
datetime模块
# 其中days = -2,可以根据需要进行替换,这样就可以得到不同需要的日期了. # # 另外:可以通过strftime方法,指定时间的输出格式. # # 除了以上输入的 %Y-%m-%d ...

记一次网络原因导致的mysql连接中断问题(druid)

记一次网络原因导致的mysql连接中断问题(druid)的更多相关文章

随机推荐

热门专题