生产环境服务CPU高问题分析

问题描述：

现网个别时候会出现CPU突然飙高的现象，飙高后不能恢复正常。

分析过程：

CPU飙高后抓dump，最好本机看，其它机器看dump可能需要下载服务运行机器的sos，clr

 
 

0:000> !threadpool

The version of SOS does not match the version of CLR you are debugging.  Please

load the matching version of SOS for the version of CLR you are debugging.

CLR Version: 4.0.30319.276

SOS Version: 4.0.30319.17929

CPU utilization: 100%

Worker Thread: Total: 54 Running: 54 Idle: 0 MaxLimit: 32767 MinLimit: 8

Work Request in Queue: 0

--------------------------------------

Number of Timers: 3

--------------------------------------

Completion Port Thread:Total: 10 Free: 10 MaxFree: 16 CurrentLimit: 8 MaxLimit: 1000 MinLimit: 8

CPU确实高，54个线程都在跑着。

接着看线程都在干什么

0:000> ~*e!clrstack

OS Thread Id: 0x3278 (57)

Child SP         IP               Call Site

000000000c72cea8 0000000077ef047a [RedirectedThreadFrame: 000000000c72cea8]

000000000c72cf40 000006448056c07e ********************.

SecurityQuota.ConcurrentLinkedQueue`1[[System.__Canon, mscorlib]].TryDequeue(System.__Canon ByRef)

000000000c72cff0 00000644805999e8 ********************.SecurityQuota.SecurityFeatureDigest.GetIncrementalDigest(********************)

……

 
 

有50个线程在做该操作，review该处代码

为了防止本地代码和服务器不一致，直接反编译的服务器的代码，GetIncrementalDigest方法如下

public class SecurityFeatureDigest : ISecurityFeatureDigest

{

    // Fields

    private static Database _data = DatabaseManager.GetDatabase("SQSDB");

    private static readonly ITracing _tracer = TracingManager.GetTracing(typeof(SecurityFeatureDigest));

    private static ConcurrentLinkedQueue<List<FeatureDigest>> queue =

new ConcurrentLinkedQueue<List<FeatureDigest>>(new List<FeatureDigest>());     (2)

    private static readonly string[] SetFeatureDigestParameter = new string[] { "@digest", "@operation", "@sample", "@stamp" };

 
 

    // 略…

 
 

    public void GetIncrementalDigest(RpcServerContext context)

    {

        DateTime args = context.GetArgs<DateTime>();

        List<FeatureDigest> results = new List<FeatureDigest>();

        List<FeatureDigest> result = null;

        while (true)

        {

            if (queue.TryDequeue(out result))      (1)

            {

                List<FeatureDigest> list3 = new List<FeatureDigest>();

                list3.AddRange(result);

                queue.Enqueue(result);

                foreach (FeatureDigest digest in list3)

                {

                    if (digest.Version > args)

                    {

                        digest.Sample = string.Empty;

                        results.Add(digest);

                    }

                }

                context.Return<List<FeatureDigest>>(results);

                return;

            }

        }

    }

 
 

    // 略…

 
 

}

从抓的dump看，极有可能是死循环了。通过以上代码看到，如果(1)处的返回为false的话，则改段代码会陷入死循环。接着看下什么情况下返回false。

public class ConcurrentLinkedQueue<T>

{

    // Fields

    private Node<T, T> _head;

    private Node<T, T> _tail;

 
 

    // 略…

   
 

    public bool TryDequeue(out T result)

    {

        while (true)

        {

            Node<T, T> comparand = this._head;

            Node<T, T> node2 = this._tail;

            Node<T, T> next = comparand.Next;

            if (comparand == this._head)

            {

                if (next == null)       (3)

                {

                    result = default(T);

                    return false;

                }

                if (comparand == node2)

                {

                    Interlocked.CompareExchange<Node<T, T>>(ref this._tail, next, node2);

                }

                else

                {

                    result = next.Item;

                    if (Interlocked.CompareExchange<Node<T, T>>(ref this._head, next, comparand) == comparand)

                    {

                        break;

                    }

                }

            }

        }

        return true;

    }

从(3)处可以看出，如果head的next为null的话，则返回false。为了确认分析是否正确，看下此处的值是不是null

0:000> !dumpheap -stat

Statistics:

              MT    Count    TotalSize Class Name

000006448054b2d0        1           32 ********************.SecurityQuota.ConcurrentLinkedQueue`1

[[System.Collections.Generic.List`1[[********************.FeatureDigest, IICSecurityLibrary]], mscorlib]]

0:000> !dumpheap – -mt 000006448054b2d0   

------------------------------

Heap 6

         Address               MT     Size

00000001404836f0 000006448054b2d0       32    

total 0 objects

Statistics:

              MT    Count    TotalSize Class Name

           32 ********************.SecurityQuota.ConcurrentLinkedQueue`1

[[System.Collections.Generic.List`1[[********************.FeatureDigest, *****]], mscorlib]]

Total 1 objects

0:000> !do 00000001404836f0

Name:        ********************.SecurityQuota.ConcurrentLinkedQueue`1[[System.Collections.Generic.List`1

[[********************.FeatureDigest, *****]], mscorlib]]

MethodTable: 000006448054b2d0

EEClass:     0000064480557a90

Size:        32(0x20) bytes

File:        ********************\SecurityQuotaService.exe

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

        8 ...Canon, mscorlib]]  0 instance 00000000c3135ec8 _head

       10 ...Canon, mscorlib]]  0 instance 00000000c3135ec8 _tail

0:000> !do 00000000c3135ec8

Name:        ********************.SecurityQuota.ConcurrentLinkedQueue`1+Node`1[[System.Collections.Generic.List`1[[********************.

FeatureDigest, *****]], mscorlib],[System.Collections.Generic.List`1[[********************.FeatureDigest, *****]], mscorlib]]

MethodTable: 000006448054b7f8

EEClass:     00000644805580b8

Size:        32(0x20) bytes

File:        ********************\SecurityQuotaService.exe

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

        8       System.__Canon  0 instance 00000001404836a8 Item

       10 ...Canon, mscorlib]]  0 instance 0000000000000000 Next

发现Next为null

 
 

问题结论：

由于一些异常逻辑导致程序死循环，此处代码也有问题，对异常逻辑处理不够。

 
 

 

作者:No.40

Blog:http://www.cnblogs.com/no40