Druid出现DruidDataSource - recyle error - recyle error java.lang.InterruptedException: null异常排查与解决

一、问题回顾

线上的代码之前运行的都很平稳，突然就出现了一个很奇怪的问题，看错误信息是第三方框架Druid报出来了，连接池回收连接时出现的问题。

2018-05-14 20:01:32.810 ERROR [hystrix-UpgradeResultReportController-49][DruidDataSource.java:1297] - recyle error
java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_131]
        at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_131]
        at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1279) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:292) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4534) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.filter.FilterAdapter.dataSource_releaseConnection(FilterAdapter.java:2717) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4530) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.filter.stat.StatFilter.dataSource_releaseConnection(StatFilter.java:646) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4530) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.pool.DruidPooledConnection.syncClose(DruidPooledConnection.java:269) [druid-1.0.11.jar!/:1.0.11]
        at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:228) [druid-1.0.11.jar!/:1.0.11]
        at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:341) [spring-jdbc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
        at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:328) [spring-jdbc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
        at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:294) [spring-jdbc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
        at org.mybatis.spring.transaction.SpringManagedTransaction.close(SpringManagedTransaction.java:127) [mybatis-spring-1.3.1.jar!/:1.3.1]
        at org.apache.ibatis.executor.BaseExecutor.close(BaseExecutor.java:90) [mybatis-3.4.2.jar!/:3.4.2]
        at org.apache.ibatis.executor.CachingExecutor.close(CachingExecutor.java:64) [mybatis-3.4.2.jar!/:3.4.2]
        at org.apache.ibatis.session.defaults.DefaultSqlSession.close(DefaultSqlSession.java:264) [mybatis-3.4.2.jar!/:3.4.2]
        at org.mybatis.spring.SqlSessionUtils.closeSqlSession(SqlSessionUtils.java:193) [mybatis-spring-1.3.1.jar!/:1.3.1]
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:454) [mybatis-spring-1.3.1.jar!/:1.3.1]
        at com.sun.proxy.$Proxy133.update(Unknown Source) [?:?]
        at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:294) [mybatis-spring-1.3.1.jar!/:1.3.1]
        at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) [mybatis-3.4.2.jar!/:3.4.2]
        at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59) [mybatis-3.4.2.jar!/:3.4.2]
        at com.sun.proxy.$Proxy148.updDeviceUpgradeStatus(Unknown Source) [?:?]
        at com.phicomm.smarthome.ota.service.impl.UpgradeHistoryDaoServiceImpl.updUpgradeSuccessStatus(UpgradeHistoryDaoServiceImpl.java:50R [classes!/:0.0.1-SNAPSHOT]
        at com.phicomm.smarthome.ota.service.impl.UpgradeHistoryDaoServiceImpl$$FastClassBySpringCGLIB$$9b468fba.invoke(<generated>) [classes!/:0.0.1-SNAPSHOT]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) [spring-aop-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]

二、排查问题

查看一下他的源码

看了一下也没有发现所以然，然后去GitHub上的Druid官方开源处，看了一下历史问题修复，发现这个是旧版本已知的一个Bug。

https://github.com/alibaba/druid/issues/785

看了一下当前使用的Druid的版本【1.0.11】，然后又确认了一下官方最新的版本是【1.1.9】，已经差了很多个版本了。

果断将版本更新至最新

<dependency>
	<groupId>com.alibaba</groupId>
	<artifactId>druid</artifactId>
	<version>1.1.9</version>
</dependency>

翻看之前出错的那里的代码发现

更新到最新版本后，上述问题得到解决。

数据库连接被中断的原因有很多，有一种是初始化配置时设置的数据库连接回收时长，通过datasource.getConnontion() 取得的连接必须在removeAbandonedTimeout这么多秒内调用close(),否则就主动将其杀死，就是conn不能超过指定的租期。

详细配置可以参考这篇文章：

https://my.oschina.net/haogrgr/blog/224010