记一次网络原因导致的mysql连接中断问题(druid)

date: 2018-04-19 21:00
tag: java,mysql,exception,mat,调试,jvm
工具: gceasy.io, MAT

线上系统出现一个诡异的bug，通过heap dump分析

分析:

通过日志确认系统在一天前就已经停止运行
代码较简单应该不存在DB写入操作死锁
mysql操作不是特别频繁

定位:

使用jmap -dump导出线上的应用的heap dump
jstack 导出堆栈信息
分析jstack发现多个Thread在DB写入时在等待同一把锁
通过MAT分析dump
通过OQL在MAT中定位上面出现的锁住多个Thread的锁对象
通过分析发现锁对象的所有者是阿里druid线程池的com.alibaba.druid.pool.DruidDataSource
在IDE中查看DruidDataSource的源码发现几个关键属性: poolingCount, errorCount, destroyCount, lastError
google搜索lastError的描述Communications link failure, 得到可能是网络原因导致的mysql连接关闭
在MAT中查看poolingCount, errorCount, destroyCount属性的值发现与上述结论一致
重启应用，问题消失

思考:

为什么Druid线程池没有尝试重新连接?
需要优化日志，提高排查效率
增加必要的Metric和预警规则

程序假死时的heap dump:

OQL:

 SELECT s.poolingCount,

 s.errorCount.toString() AS errorCount,

 s.destroyCount.toString() AS destroyCount,

 s.createCount.toString() AS createCount,

 s.lastError.exceptionMessage.toString()

 FROM com.alibaba.druid.pool.DruidDataSource s

结果:

 s.poolingCount |errorCount|destroyCount|createCount|s.lastError.exceptionMessage.toString()

---------------------------------------------------------------------------------------------------------------------------------------------

              0 |10        |7           |17         |Communications link failure\u000a\u000aLast packet sent to the server was 180289 ms ago.

---------------------------------------------------------------------------------------------------------------------------------------------

s.connections:

结果:

Type|Name|Value

---------------

ref |[0] |null

ref |[1] |null

ref |[2] |null

ref |[3] |null

ref |[4] |null

ref |[5] |null

ref |[6] |null

ref |[7] |null

---------------

正常运行时候的core dump:

OQL:

 SELECT s.poolingCount,

 s.errorCount.toString() AS errorCount,

 s.destroyCount.toString() AS destroyCount,

 s.createCount.toString() AS createCount,

 s.lastError.exceptionMessage.toString()

 FROM com.alibaba.druid.pool.DruidDataSource s

结果:

 s.poolingCount |errorCount|destroyCount|createCount|s.lastError.exceptionMessage.toString()

--------------------------------------------------------------------------------------------

              1 |1         |0           |4          |

--------------------------------------------------------------------------------------------

s.connections:

结果:

Type|Name|Value

-------------------------------------------------------------------

ref |[0] |com.alibaba.druid.pool.DruidConnectionHolder @ 0xc5fb1958

ref |[1] |null

ref |[2] |null

ref |[3] |null

ref |[4] |null

ref |[5] |null

ref |[6] |null

ref |[7] |null

-------------------------------------------------------------------

日志文件中的异常堆栈:

org.springframework.dao.RecoverableDataAccessException:

### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

### The error may exist in class path resource [mapper/ConfigMapper.xml]

### The error may involve com.myserver1.crawler.dao.ConfigDao.selectAllByType-Inline

### The error occurred while setting parameters

### SQL: select           id, type, props, content, status, create_time, update_time         from config     where     type = ?     and status = 1

### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

; SQL []; Communications link failure

Last packet sent to the server was 655505 ms ago.; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

        at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:98) ~[spring-jdbc-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:74) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:399) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        at com.sun.proxy.$Proxy60.selectList(Unknown Source) ~[?:?]

        at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:205) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        at com.myserver1.crawler.dao.ConfigDao.selectAllByType(ConfigDao.java:27) ~[classes!/:1.0-SNAPSHOT]

        at com.myserver1.crawler.dao.ConfigDao$$FastClassBySpringCGLIB$$98d41dde.invoke(<generated>) ~[classes!/:1.0-SNAPSHOT]

        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.3.6.RELEASE.jar!/:4.3.6.RELEAS

E]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) ~[spring-aop-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]

        at com.myserver1.crawler.dao.ConfigDao$$EnhancerBySpringCGLIB$$c890af99.selectAllByType(<generated>) ~[classes!/:1.0-SNAPSHOT]

        at com.myserver1.event.VideoItemListener.post(VideoItemListener.java:32) ~[classes!/:1.0-SNAPSHOT]

        at com.myserver1.event.EventBus.push(EventBus.java:28) [classes!/:1.0-SNAPSHOT]

        at com.myserver1.crawler.site.ABreadthCrawler.start(ABreadthCrawler.java:97) [classes!/:1.0-SNAPSHOT]

        at com.myserver1.event.QueueThread.run(QueueThread.java:41) [classes!/:1.0-SNAPSHOT]

Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 655505 ms ago.

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_101]

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_101]

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_101]

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_101]

        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3246) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:477) ~[druid-0.2.9.jar!/:0.2.9]

        at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:78) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113) ~[mybatis-3.3.0.jar!/:3.3.0]

        at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_101]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]

        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:386) ~[mybatis-spring-1.2.3.jar!/:1.2.3]

        ... 15 more

Caused by: java.net.SocketException: Connection reset

        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[?:1.8.0_101]

        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_101]

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_101]

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_101]

        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar!/:?]

        at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:477) ~[druid-0.2.9.jar!/:0.2.9]

        at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:78) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120) ~[mybatis-3.3.0.jar!/:3.3.0]

        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113) ~[mybatis-3.3.0.jar!/:3.3.0]

        at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source) ~[?:?]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_101]

记一次网络原因导致的mysql连接中断问题(druid)的更多相关文章

网络原因导致 npm 软件包 node-sass / gulp-sass 安装失败的处理办法
如果你正在构建一个基于 gulp 的前端自动化开发环境,那么极有可能会用到 gulp-sass ,由于网络原因你可能会安装失败,因为安装过程中部分细节会到亚马逊云服务器上获取文件.本文主要讨论在不变更 ...
网络原因导致的 spring cloud config 读取git上的配置文件时报错：Cannot clone or checkout repository
今天在公司使用spring cloud config搭建配置中心的时候,出现了读取不到git库的问题:Cannot clone or checkout repository.在网上百度,前面几个答案都 ...
记一次低级错误导致的mysql（111）
今天下午配好的双主多从服务器,两台主机+主机内安装好的6台虚拟机,两台Mysql master各授权好其slave的远程登录,原本好端端的能远程登录,晚上回来时候就发现其中一台master登录不上其s ...
Java Mysql连接池配置和案例分析--超时异常和处理
前言: 最近在开发服务的时候, 发现服务只要一段时间不用, 下次首次访问总是失败. 该问题影响虽不大, 但终究影响用户体验. 观察日志后发现, mysql连接因长时间空闲而被关闭, 使用时没有死链检测 ...
MongoDB3.4安装配置以及与Robomongo1.1的连接——解决Authentication Failed导致的不能连接问题
本文环境:win10(64)+MongoDB(3.4.5)+Robomongo(1.1) 目录: MongoDB的安装 MongoDB的配置 Robomongo的安装以及与MongoDB的连接一些新 ...
关于流量升高导致TIME_WAIT增加，MySQL连接大量失败的问题
有个应用就是每次都会去查一个接口,接口返回用户的信息数据,从而展现不同的页面效果.大致流程如下应用APP(电信)-> memcache ->电信custom接口 ->master- ...
SQLServer 2012之AlwaysOn —— 指定数据同步链路，消除网络抖动导致的提交延迟问题
事件起因:近期有研发反应,某数据库从08切换到12环境后,不定期出现写操作提交延迟的问题: 事件分析:在排除了系统资源争用等问题后,初步分析可能由于网络抖动导致同步模式alwayson节点经常出现会话 ...
记录一次mongodb因网络问题导致shard节点异常
现象: 机房反馈9点左右,机房交换机故障,导致网络出现问题业务人员反馈某个接口超时初查:通过业务日志查看分析发现,在连接mongo的某个collections时候,报错错误如下: 在写入数据的时候 ...
electron-builder 由于网络原因无法下载问题解决
electron-builder 由于网络原因无法下载问题解决在package.json的build中添加electron的镜像 "electronDownload": { &q ...

随机推荐

总结get和post区别---面试用
总结get和post区别---面试用我是搬运工!!!! get参数通过url传递,post放在request body中. get请求在url中传递的参数是有长度限制的,而post没有. get比p ...
【webpack学习笔记】a07-代码分离
官方文档说进行代码分离有三种方法: 入口起点:使用entry配置手动分离. 防止重复:使用CommonsChunkPlugin插件去重合分离chunk 注:在webpack4中,CommonsChun ...
微信小程序-解决下拉刷新报错
关于“enablePullDownRefresh”: “true” 一.使用方式在 Page 中定义 onPullDownRefresh 处理函数,监听该页面用户下拉刷新事件.需要在 config ...
perceptual loss
https://arxiv.org/abs/1603.08155 两个网络:image transfer网络和loss网络 image transfer网络: 将输入图片y通过映射f W (x)得到输 ...
前端车牌识别SDK算法提取
同行业中,别人标配有的产品我有,别人没有的产品我们也有,如此才能增强竞争力,通过优化创新,前端车牌识别SDK功能,性能上,都是行业NO.1的水平.车牌识别sdk这个用于越来越多人集成了,汽车保有量日益 ...
css——letter spacing
letter spacing属性是用来设置字母的间距实例: 设置h1和h2之间字母的间距 <html> <head> <style type="text/cs ...
eureka注册中心列表页面加账号和密码
正常情况下,注册中心服务启动后,页面是这样的,如下图所示这样的话,如果注册中心不再内网地址下,有可能就会泄露所有的服务地址信息,增加受攻击的风险,针对这个问题咱们可以这样解决首先注册中心pom.x ...
react系列笔记：第一记-redux
前言: 目前公司使用dva,对于前不久还是使用原生js的我来说,花了差不多一两周时间,基本掌握如何使用.虽然对于react有一点点基础,但很多地方未深入,很多概念也很模糊,故从本文开始,记录一下系统的 ...
P1403 [AHOI2005]约数研究题解
转载luogu某位神犇的题解QAQ 这题重点在于一个公式: f(i)=n/i 至于公式是怎么推出来的,看我解释: 1-n的因子个数,可以看成共含有2因子的数的个数+含有3因子的数的个数……+含有n因子 ...
logging dictconfig
#coding: utf-8 import logging import logging.config class SingleLevelFilter(object): def __ini ...

记一次网络原因导致的mysql连接中断问题(druid)

记一次网络原因导致的mysql连接中断问题(druid)的更多相关文章

随机推荐

热门专题