由dubbo服务禁用system.gc而引起的思考

我一直都有一个疑问，丰巢业务服务的生产环境jvm参数设置是禁止system.gc的，也就是开启设置：-XX:+DisableExplicitGC，但是生产环境却从来没有出现过堆外内存溢出的情况。说明一下，丰巢使用了阿里开源的dubbo，而dubbo底层通信默认情况下使用了3.2.5.Final版本的netty，而我们对于netty的常规认知里，netty一定是使用了堆外内存，并且堆外内存在禁止了system.gc这个函数调用的话，在服务没有主动回收分配的堆外内存的情况下，一定会出现堆外内存的泄露。带着这个问题，刚好前天晚上有些时间，研究了一下3.2.5版本的netty源码，又是在科兴科兴园等馒头妈妈时候，发现了秘密之所在，我只能说，科兴科学园真是我的宝地啊。

 

涉及到的netty类：NioWorker、HeapChannelBufferFactory、BigEndianHeapChannelBuffer、SocketReceiveBufferPool

 

核心的秘密在SocketReceiveBufferPool中

 final class SocketReceiveBufferPool {

     private static final int POOL_SIZE = 8;

     @SuppressWarnings("unchecked")
     private final SoftReference<ByteBuffer>[] pool = new SoftReference[POOL_SIZE];

     SocketReceiveBufferPool() {
         super();
     }

     final ByteBuffer acquire(int size) {
         final SoftReference<ByteBuffer>[] pool = this.pool;
         for (int i = 0; i < POOL_SIZE; i ++) {
             SoftReference<ByteBuffer> ref = pool[i];
             if (ref == null) {
                 continue;
             }

             ByteBuffer buf = ref.get();
             if (buf == null) {
                 pool[i] = null;
                 continue;
             }

             if (buf.capacity() < size) {
                 continue;
             }

             pool[i] = null;

             buf.clear();
             return buf;
         }

         ByteBuffer buf = ByteBuffer.allocateDirect(normalizeCapacity(size));
         buf.clear();
         return buf;
     }

     final void release(ByteBuffer buffer) {
         final SoftReference<ByteBuffer>[] pool = this.pool;
         for (int i = 0; i < POOL_SIZE; i ++) {
             SoftReference<ByteBuffer> ref = pool[i];
             if (ref == null || ref.get() == null) {
                 pool[i] = new SoftReference<ByteBuffer>(buffer);
                 return;
             }
         }

         // pool is full - replace one
         final int capacity = buffer.capacity();
         for (int i = 0; i< POOL_SIZE; i ++) {
             SoftReference<ByteBuffer> ref = pool[i];
             ByteBuffer pooled = ref.get();
             if (pooled == null) {
                 pool[i] = null;
                 continue;
             }

             if (pooled.capacity() < capacity) {
                 pool[i] = new SoftReference<ByteBuffer>(buffer);
                 return;
             }
         }
     }

     private static final int normalizeCapacity(int capacity) {
         // Normalize to multiple of 1024
         int q = capacity >>> 10;
         int r = capacity & 1023;
         if (r != 0) {
             q ++;
         }
         return q << 10;
     }
 }

SocketReceiveBufferPool中维护了一个SoftReference<ByteBuffer>类型的数组，关于java的SoftReference，大家可以自行搜索。其实就是在此类中维护了一个directbuffer的内存池，此部分的内存是可以重复利用的。那么问题来了，如果我们把netty用于接收网络信息的directbuffer直接传给dubbo的业务代码，那么这个内存池的作用是什么呢，内存如何被release回内存池？带着这个疑问，继续分析调用了SocketReceiveBufferPool的NioWorker代码。

     private boolean read(SelectionKey k) {
         final SocketChannel ch = (SocketChannel) k.channel();
         final NioSocketChannel channel = (NioSocketChannel) k.attachment();

         final ReceiveBufferSizePredictor predictor =
             channel.getConfig().getReceiveBufferSizePredictor();
         final int predictedRecvBufSize = predictor.nextReceiveBufferSize();

         int ret = 0;
         int readBytes = 0;
         boolean failure = true;

         ByteBuffer bb = recvBufferPool.acquire(predictedRecvBufSize);
                  try {
             while ((ret = ch.read(bb)) > 0) {
                 readBytes += ret;
                 if (!bb.hasRemaining()) {
                     break;
                 }
             }
             failure = false;
         } catch (ClosedChannelException e) {
             // Can happen, and does not need a user attention.
         } catch (Throwable t) {
             fireExceptionCaught(channel, t);
         }

         if (readBytes > 0) {
             bb.flip();

             final ChannelBufferFactory bufferFactory =
                 channel.getConfig().getBufferFactory();
             final ChannelBuffer buffer = bufferFactory.getBuffer(readBytes);
             buffer.setBytes(0, bb);
             buffer.writerIndex(readBytes);
             //if(buffer instanceof BigEndianHeapChannelBuffer){
             //    logger2.info("buffer instanceof BigEndianHeapChannelBuffer.");
             //}
             recvBufferPool.release(bb);

             // Update the predi||\\|||||
             predictor.previousReceiveBufferSize(readBytes);

             // Fire the event.
             fireMessageReceived(channel, buffer);
         } else {
             recvBufferPool.release(bb);
         }

         if (ret < 0 || failure) {
             k.cancel(); // Some JDK implementations run into an infinite loop without this.
             close(channel, succeededFuture(channel));
             return false;
         }

         return true;
     }

在代码里发现了netty会再创造一个chanelbuffer对象，然后将directbuffer里的内容复制到chanelbuffer里面，而这个chanelbuffer对象实际上是一个堆内内存，然后netty会真对这块内存进行解码及返回给上层调用服务等，也就是说没有直接将directbuffer返回给dubbo服务，这样也就解释了，我们在提供dubbo服务的jvm里，禁止掉了system.gc的情况下，没有发生过堆外内存泄漏的原因。后面我会找时间详细的分析一下netty4和kafka使用directbuffer的情况。