proxool 连接池警告分析：appears to have started a thread named [HouseKeeper] but has failed to stop it

1. 问题：日志中出现下面的警告：

警告: The web application [ROOT] appears to have started a thread named [HouseKeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
org.logicalcobwebs.proxool.HouseKeeperThread.run(HouseKeeperThread.java:46)

2. 原因调查：

使用的是 proxool 连接池，查看源码，分析如下：

在项目某种情况下，ProxoolFacade.shutdown(); 被调用，他会调用：

    protected static void shutdown(String finalizer, int delay) {

        ConnectionPool[] cps = ConnectionPoolManager.getInstance().getConnectionPools();
        for (int i = 0; i < cps.length; i++) {
            removeConnectionPool(finalizer, cps[i], delay);
        }

        // If a shutdown hook was registered then remove it
        try {
            if (shutdownHook != null) {
                ShutdownHook.remove(shutdownHook);
            }
        } catch (Throwable t) {
            if (LOG.isDebugEnabled()) {
                LOG.debug("Unanticipated error during removal of ShutdownHook. Ignoring it.", t);
            }
        }

        // Stop threads
        PrototyperController.shutdown();
        HouseKeeperController.shutdown();

    }

这里 首先 关闭连接池，然后在最后面关闭了一个 housekeeper 的 demaon线程：

 HouseKeeperController.shutdown();

    /**
     * Stop all house keeper threads.
     */
    protected static void shutdown() {
        synchronized(LOCK) {
            Iterator i = houseKeeperThreads.iterator();
            while (i.hasNext()) {
                HouseKeeperThread hkt = (HouseKeeperThread) i.next();
                LOG.info("Stopping " + hkt.getName() + " thread");
                hkt.cancel();
            }
            houseKeeperThreads.clear();
        }
    }

对每一个 housekeeper 线程调用 cancel() 方法，一般只有一个 housekeeper线程：

hkt.cancel();

public class HouseKeeperThread extends Thread {
    private static final Log LOG = LogFactory.getLog(HouseKeeperThread.class);
    private boolean stop;
    public HouseKeeperThread(String name) {
        setDaemon(true);
        setName(name);
    }
    public void run() {
        while (!stop) {
            HouseKeeper hk = HouseKeeperController.getHouseKeeperToRun();
            while (hk != null && !stop) {
                try {
//                    if (LOG.isDebugEnabled()) {
//                        LOG.debug("About to sweep " + hk.getAlias());
//                    }
                    hk.sweep();
                } catch (ProxoolException e) {
                    LOG.error("Couldn't sweep " + hk.getAlias(), e);
                }
                hk = HouseKeeperController.getHouseKeeperToRun();
            }
            try {
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                LOG.error("Interrupted", e);
            }
        }
    }
    protected void cancel() {
        stop = true;
    }
}

尼玛 stop 尽然是一个 普通 的 boolean 变量，感觉应该使用 volatile 修饰！

保错的原因是，housekeeper 线程因为每隔5秒执行一次，而系统 想要 stop 它的时候，它正在 sleep ，所以抛出了异常：

警告: The web application [ROOT] appears to have started a thread named [HouseKeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.logicalcobwebs.proxool.HouseKeeperThread.run(HouseKeeperThread.java:46)

但是，因为 设置了 stop = true，所以，当它 sleep 完成之后，也可能会 判断 stop == true, 也可能判断 stop == false，因为这里没有使用 volatile 修饰！

所以最坏的结果，就是 有一个名叫 "HouseKeeper" 的 deamon jvm 线程没有被成功关闭。

3. 日志显示，数据库连接池，确实被正确关闭了：

ContainerBackgroundProcessor[StandardEngine[Catalina]] - Proxool shoutdown
[INFO ] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:484):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Shutting down 'autumn' pool immediately [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/05/00) - #0006 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #6 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/04/00) - #0005 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #5 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/03/00) - #0004 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #4 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/02/00) - #0003 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #3 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/01/00) - #0002 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #2 closed
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/00/00) - #0001 removed because of shutdown.
[DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #1 closed
[INFO ] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:564):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 'autumn' pool has been closed down by ContainerBackgroundProcessor[StandardEngine[Catalina]] in 6 milliseconds.
[DEBUG] org.logicalcobwebs.proxool.ShutdownHook.remove(ShutdownHook.java:41):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Removed shutdownHook
[INFO ] org.logicalcobwebs.proxool.PrototyperController.shutdown(PrototyperController.java:100):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Stopping Prototyper thread
[INFO ] org.logicalcobwebs.proxool.HouseKeeperController.shutdown(HouseKeeperController.java:107):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Stopping HouseKeeper thread

排除了 poxool 导致 数据库连接泄露的问题。