利用windbg 分析IIS 的线程池w3wp程序多线程挂起问题

前几天有个朋友发个了在windows server 2008跑的IIS 跑的程序w3wp程序dmp,要我帮忙分析为何线程都挂起不运行

经过查阅资料用windbg可以调试可以输出线程的调用堆栈,但是准备调试之前费了不少功夫和时间

主要碰到是SOS.DLL和DMP 文件运行时候的CLR.DLL不匹配问题,那就需要从w3wp运行的机器上拷贝到对应.net版本下面的sos.dll,clr.dll,mscordacwks.dll文件

然后给windbg 加载,主要是通过.load 命令,卸载扩展DLL命令是.unload ,查看当前加载的DLL命令是.chain

还需要设置好windbg 的调试符号

下载好符号后，输入下面命令，没出现文件不匹配提示就表示可以使用SOS.dll的扩展命令

0:000> !clrstack
OS Thread Id: 0x1cc4 (0)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
Failed to start stack walk: 80070057

如果出现下面情况，就是文件不匹配，你需要具体查看加载的SOS.DLL文件版本,
0:033> !clrstack

The version of SOS does not match the version of CLR you are debugging.  Please

load the matching version of SOS for the version of CLR you are debugging.

CLR Version: 4.0.30319.1008

SOS Version: 4.0.30319.1022

有时候运行机器未必 能找到对应版本的SOS.dll 你就必须要到微软官网上面找到对应系统补丁的里面的.NET版本SOS.DLL

解压出微软补丁的工具msix 具体怎么用百度或者谷歌下

查阅线程  !runaway 或者!threads

0:000> !runaway
User Mode Time
Thread Time
52:13b0 0 days 0:00:23.421
41:1124 0 days 0:00:18.359
42:1cf4 0 days 0:00:11.687
44:1d64 0 days 0:00:09.390
14:13ec 0 days 0:00:07.375
16:1c08 0 days 0:00:07.171
19:1d60 0 days 0:00:06.640
18:1cc8 0 days 0:00:06.640
15:f64 0 days 0:00:06.437
17:f88 0 days 0:00:06.375

经过查阅资料 查询所有线程调用堆栈~*e !clrstack 或者!EEStack -EE 后者的命令打印比较好看些

0:000> ~*e !clrstack

OS Thread Id: 0x954 (48)
Child SP IP Call Site
0000000011a3e538 000000007793c0ea [GCFrame: 0000000011a3e538]
0000000011a3e608 000000007793c0ea [HelperMethodFrame_1OBJ: 0000000011a3e608] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
0000000011a3e720 000007fe99aef61f Disruptor.BlockingWaitStrategy.WaitFor(Int64, Disruptor.Sequence, Disruptor.Sequence[], Disruptor.ISequenceBarrier)
0000000011a3e780 000007fe99aef548 Disruptor.ProcessingSequenceBarrier.WaitFor(Int64)
0000000011a3e7c0 000007fe99aeefcb Disruptor.BatchEventProcessor`1[[System.__Canon, mscorlib]].Run()
0000000011a3e840 000007fef7b3c1de System.Threading.Tasks.Task.Execute()
0000000011a3e8b0 000007fef7bed0b5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0000000011a3ea10 000007fef7bece19 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0000000011a3ea40 000007fef7b3bf10 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
0000000011a3eb30 000007fef7b3bca5 System.Threading.Tasks.Task.ExecuteEntry(Boolean)
0000000011a3eb70 000007fef7b2764a System.Threading.ThreadPoolWorkQueue.Dispatch()
0000000011a3f0c8 000007fef90b9e03 [DebuggerU2MCatchHandlerFrame: 0000000011a3f0c8]
0000000011a3f2a8 000007fef90b9e03 [ContextTransitionFrame: 0000000011a3f2a8]
0000000011a3f498 000007fef90b9e03 [DebuggerU2MCatchHandlerFrame: 0000000011a3f498]

0:000> !EEStack -EE

Thread 42
Current frame:
Child-SP RetAddr Caller, Callee
00000000171fd990 000007fef4402108 (MethodDesc 000007fef4132438 +0x68 System.Threading.ExecutionContextSwitcher.Undo())
00000000171fd9e0 000007fef4401c0f (MethodDesc 000007fef40104e8 +0x19f System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000171fdd20 000007fef43e8ce0 (MethodDesc 000007fef4024840 +0xe0 System.Text.StringBuilder.Append(System.String))
00000000171fdf00 000007fef51475e2 (MethodDesc 000007fef400d340 System.Runtime.Remoting.Messaging.LogicalCallContext.Clone())
00000000171fdf50 000007fef4449282 (MethodDesc 000007fef413cd18 +0x302 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken))
00000000171fe030 000007fef4449282 (MethodDesc 000007fef413cd18 +0x302 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken))
00000000171fe0c0 000007fef4441bf7 (MethodDesc 000007fef4016870 +0xb7 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken))
00000000171fe130 000007fef4d89942 (MethodDesc 000007fef4016860 +0x222 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken))
00000000171fe150 000007fef1c4c88d (MethodDesc 000007fef1c31d58 +0x3d System.Net.Http.HttpUtilities.ContinueWithStandard[[System.__Canon, mscorlib]](System.Threading.Tasks.Task`1<System.__Canon>, System.Action`1<System.Threading.Tasks.Task`1<System.__Canon>>))
00000000171fe220 000007fef4d86687 (MethodDesc 000007fef402d0a8 +0x27 System.Threading.Tasks.Task`1[[System.__Canon, mscorlib]].GetResultCore(Boolean))
00000000171fe260 000007fe9ac40c0b (MethodDesc 000007fe9a085250 +0x1cb WebMvc.App_Start.FileUploadSDK+<>c.<UploadHTML>b__11_0(System.Object))
00000000171fe350 000007fef4401bce (MethodDesc 000007fef40104e8 +0x15e System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000171fe420 000007fef4401a67 (MethodDesc 000007fef40104d8 +0x17 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
00000000171fe430 000007fef43c5eb5 (MethodDesc 000007fef4152010 +0x2e5 System.Threading.ThreadPoolWorkQueue.Dispatch())
00000000171fe450 000007fef439c0a0 (MethodDesc 000007fef402f670 +0x70 System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem())
00000000171fe490 000007fef43c5d30 (MethodDesc 000007fef4152010 +0x160 System.Threading.ThreadPoolWorkQueue.Dispatch())
00000000171fe7f0 000007fef3290a2e (MethodDesc 000007fef2ff8d38 +0x7e DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef))
00000000171fe850 000007fef322720e (MethodDesc 000007fef2fd7d90 +0x65e System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32))
00000000171fea00 000007fef3226b84 (MethodDesc 000007fef2fd7d80 +0x14 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32))
---------------------------------------------

参考资料

https://msdn.microsoft.com/en-us/library/bb190764(v=vs.110).aspx?query=

http://www.cnblogs.com/softfair/p/The_version_SOS_not_match_version_of_CLR_PDB_symbol_for_clr_dll_not_loaded.html

https://www.cnblogs.com/zrhai/p/3784206.html