记一次 .NET某培训学校系统 内存碎片化分析

一：背景

1. 讲故事

前些天有位朋友微信上找到我，说他们学校的Web系统内存一直下不去，让我看下到底是怎么回事，老规矩让朋友生成一个dump文件丢给我，看一下便知。

二：WinDbg 分析

1. 托管还是非托管

要想看托管还是非托管，可以用 !address -summary 观察下内存段。

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    384     7dbe`3d662000 ( 125.743 TB)           98.24%
<unknown>                              9653      241`9e9fd000 (   2.256 TB)  99.98%    1.76%
Stack                                   153        0`125d0000 ( 293.812 MB)   0.01%    0.00%
Image                                  1912        0`0b0f2000 ( 176.945 MB)   0.01%    0.00%
Heap                                    110        0`0669a000 ( 102.602 MB)   0.00%    0.00%
Other                                    12        0`001ce000 (   1.805 MB)   0.00%    0.00%
TEB                                      51        0`00066000 ( 408.000 kB)   0.00%    0.00%
PEB                                       1        0`00001000 (   4.000 kB)   0.00%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                384     7dbe`3d662000 ( 125.743 TB)           98.24%
MEM_RESERVE                            2794      23f`666dd000 (   2.248 TB)  99.59%    1.76%
MEM_COMMIT                             9098        2`5c2b1000 (   9.440 GB)   0.41%    0.01%

从卦中信息的 MEM_COMMIT 和 Heap 来看，应该就是托管内存的问题了。

2. 托管堆排查

要查看托管堆，可以用 !eeheap -gc 观察下。

0:000> !eeheap -gc

========================================
Number of GC Heaps: 8
----------------------------------------
Heap 0 (00000273748727a0)
Small object heap
         segment            begin        allocated        committed allocated size       committed size
generation 0:
    02b388ff24f0     027615400020     0276157f76f8     027615800000 0x3f76d8 (4159192)   0x400000 (4194304)
...
generation 1:
    02b388ff5e00     02762a000020     02762a01e460     02762a0b0000 0x1e440 (123968)     0xb0000 (720896)
generation 2:
    02b388f81840     027385000020     0273853a2b20     027385400000 0x3a2b00 (3812096)   0x400000 (4194304)
    02b388f818f0     027385400020     0273857ee220     027385800000 0x3ee200 (4121088)   0x400000 (4194304)
    02b388f81e70     027387400020     0273877914e8     0273877b2000 0x3914c8 (3740872)   0x3b2000 (3874816)
    02b388f823f0     027389400020     0273897bf580     0273897df000 0x3bf560 (3929440)   0x3df000 (4059136)
    02b388f82600     02738a000020     02738a3644e0     02738a368000 0x3644c0 (3556544)   0x368000 (3571712)
    ...
    02b388f85fc0     02739f000020     02739f253e10     02739f257000 0x253df0 (2440688)   0x257000 (2453504)
    02b388f861d0     02739fc00020     02739fffc9f0     02739ffff000 0x3fc9d0 (4180432)   0x3ff000 (4190208)
    02b388f86490     0273a0c00020     0273a0f829c0     0273a0fbd000 0x3829a0 (3680672)   0x3bd000 (3919872)
    02b388f87040     0273a5000020     0273a5332f78     0273a5337000 0x332f58 (3354456)   0x337000 (3371008)
    02b388f875c0     0273a7000020     0273a72e4710     0273a7305000 0x2e46f0 (3032816)   0x305000 (3166208)
    ...
Large object heap
         segment            begin        allocated        committed allocated size       committed size
    02b388f84f40     027399000020     02739908f520     027399090000 0x8f500 (587008)     0x90000 (589824)
Pinned object heap
         segment            begin        allocated        committed allocated size       committed size
    02b388f812c0     027383000020     02738300b080     027383011000 0xb060 (45152)       0x11000 (69632)
------------------------------
GC Allocated Heap Size:    Size: 0x2212923b0 (9146278832) bytes.
GC Committed Heap Size:    Size: 0x23b676000 (9586565120) bytes.

从卦中看当前的托管内存是 9.5G，通过观察内存都是被 Gen2 给吃掉了，那 Gen2 上都是什么对象呢？我们用 !dumheap -stat 观察下。

0:000> !dumpheap -stat
Statistics:
          MT     Count     TotalSize Class Name
...
7ff954ecd918    91,030     9,467,120 System.Reflection.RuntimeMethodInfo
7ff95701c8f0   572,034    18,305,088 System.Globalization.DateTimeFormatInfo+TokenHashValue
7ff954c9fd00 1,010,024    62,198,216 System.String
7ff95502fc10    62,645   173,045,678 System.Byte[]
0273747f9610    28,313 8,761,969,520 Free
Total 3,305,047 objects, 9,143,856,580 bytes

不看卦不知道，一看吓一跳，9G的内存，Free 就吃掉了 8.7G，看样子又是经典的 内存碎片化 了，赶紧到 Gen2 上去看一看狼狈现场。

0:000> !dumpheap 02739fc00020 02739fffc9f0
         Address               MT           Size
    02739fc00020     0273747f9610        717,136 Free
    02739fcaf170     7ff95502fc10          8,216
    02739fcb1188     7ff955478798             64
    02739fcb11c8     0273747f9610        607,864 Free
    02739fd45840     7ff95502fc10          8,216
    02739fd47858     7ff955478798             64
    02739fd47898     0273747f9610      2,205,336 Free
    02739ff61f30     7ff95502fc10          8,216
    02739ff63f48     7ff955478798             64
    02739ff63f88     0273747f9610         99,736 Free
    02739ff7c520     7ff95502fc10          8,216
    02739ff7e538     7ff955478798             64
    02739ff7e578     0273747f9610         76,504 Free
    02739ff91050     7ff95502fc10          8,216
    02739ff93068     7ff955478798             64
    02739ff930a8     0273747f9610        355,728 Free
    02739ffe9e38     7ff95502fc10          8,216
    02739ffebe50     7ff955478798             64
    02739ffebe90     0273747f9610         60,168 Free
    02739fffa998     7ff95502fc10          8,216
    02739fffc9b0     7ff955478798             64 

Statistics:
          MT Count TotalSize Class Name
7ff955478798     7       448 System.IO.FileSystemWatcher+AsyncReadState
7ff95502fc10     7    57,512 System.Byte[]
0273747f9610     7 4,122,472 Free
Total 21 objects, 4,180,432 bytes
...

卦中信息是明显的 内存碎片化 现象，可以看到每一个 Free 后面都跟着一个 8216，那这玩意是干嘛的，为什么它不会 GC 回收呢？

3. 碎片化排查

要想找到不会回收的原因，我们用 !gcroot 看一下。

0:000> !gcroot 02739fffa998
Caching GC roots, this may take a while.
Subsequent runs of this command will be faster.

HandleTable:
    0000027374723fc0 (async pinned handle)
          -> 02739dc758c8     System.Threading.OverlappedData
          -> 02739fffa998     System.Byte[] 

从卦中可以看到它是被 System.Threading.OverlappedData 持有，熟悉异步编程的朋友应该都知道IO完成端口，C# 中的 OverlappedData 还会绑定 handle，byte[]，IOThread 等信息，我们挖一下其中的 _userState 。

0:000> !do 02739dc758c8
Name:        System.Threading.OverlappedData
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ff95502bce0  4000b00        8  System.IAsyncResult  0 instance 0000000000000000 _asyncResult
00007ff954af93b0  4000b01       10        System.Object  0 instance 000002739dc75910 _callback
00007ff95547f208  4000b02       18 ...eading.Overlapped  0 instance 000002739dc75880 _overlapped
00007ff954af93b0  4000b03       20        System.Object  0 instance 000002739fffa998 _userObject
00007ff954c9ac30  4000b04       28                  PTR  0 instance 000002b41023f130 _pNativeOverlapped
00007ff954c99250  4000b05       30        System.IntPtr  1 instance 0000000000000000 _eventHandle
00007ff954c1e8c0  4000b06       38         System.Int32  1 instance                0 _offsetLow
00007ff954c1e8c0  4000b07       3c         System.Int32  1 instance                0 _offsetHigh

0:000> !do 000002739dc75880
Name:        System.Threading.ThreadPoolBoundHandleOverlapped
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ff95547f558  4000b08        8 ...ng.OverlappedData  0 instance 000002739dc758c8 _overlappedData
00007ff955478d48  4000d2f       10 ...ompletionCallback  0 instance 00000273869e27a8 _userCallback
00007ff954af93b0  4000d30       18        System.Object  0 instance 000002739fffc9b0 _userState
00007ff9554791a8  4000d31       20 ...locatedOverlapped  0 instance 000002739dc75860 _preAllocated
00007ff954c9ac30  4000d32       30                  PTR  0 instance 000002b41023f130 _nativeOverlapped
00007ff955479790  4000d33       28 ...adPoolBoundHandle  0 instance 0000000000000000 _boundHandle
00007ff954c1b3c0  4000d34       38       System.Boolean  1 instance                0 _completed
00007ff955478d48  4000d2e      930 ...ompletionCallback  0   static 00000273869e2898 s_completionCallback

0:000> !do 000002739fffc9b0
Name:        System.IO.FileSystemWatcher+AsyncReadState
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ff954c1e8c0  400002b       30         System.Int32  1 instance                1 <Session>k__BackingField
00007ff95502fc10  400002c        8        System.Byte[]  0 instance 000002739fffa998 <Buffer>k__BackingField
00007ff955026688  400002d       10 ...es.SafeFileHandle  0 instance 000002739dc757d8 <DirectoryHandle>k__BackingField
00007ff955479790  400002e       18 ...adPoolBoundHandle  0 instance 000002739dc75828 <ThreadPoolBinding>k__BackingField
00007ff9554791a8  400002f       20 ...locatedOverlapped  0 instance 000002739dc75860 <PreAllocatedOverlapped>k__BackingField
00007ff955479248  4000030       28 ...eSystem.Watcher]]  0 instance 000002739dc75848 <WeakWatcher>k__BackingField

从卦中可以看到原来是 FileSystemWatcher 在作祟，说实话看到这个东西我马上就有条件反射,肯定又是经典的 ReloadOnChange=true 导致的，果然直觉就是对的，一搜代码果然有，截图如下：

三：总结

说实话 ReloadOnChange=true 真的是万恶之源，据往期分析可列出如下四大罪证：

• 文件句柄暴涨
• byte[]导致的内存暴涨
• pinned 导致的内存碎片化
• 线程池暴涨 导致的程序无响应

大家使用前需 慎之又慎，三思而后行！！！