Linux高级调试与优化——内存泄漏实战分析

最近在整理Linux调试方面的文档，正好碰到了一个内存泄漏踩栈的问题，借此机会记录一下分析过程。

首先，发现问题之后，赶紧看一下产生coredump文件没有，果不其然，产生了coredump，果断上gdb调试。

$ arm-buildroot-linux-gnueabi-gdb ./linecard ~/core_tMscRcv_165

GNU gdb (GDB) 7.10.1 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.  Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-buildroot-linux-gnueabi". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./linecard...done.

warning: exec file is newer than core file.

[New LWP 276]

... ...

[New LWP 303]

warning: .dynamic section for "./libethernet_oam.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libigmp_adapter.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libmsc.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libolt_config.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libonu_config.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libonu_vlan.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libpolicy.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libppt.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./librms.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libtime_sync.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libvoice.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libprivate_com.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libonu_ability.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libomci.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libconfig_data_gpon.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./liblineid_adapter.so" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "./libbep.so" is not at the expected address (wrong library or version mismatch?)

warning: Could not load shared library symbols for 11 libraries, e.g. /usr/local/libfhdrv_kdrv_board_impl.so.

Use the "info sharedlibrary" command to see the complete listing.

Do you need "set solib-search-path" or "set sysroot"?

Core was generated by `./linecard'.

Program terminated with signal SIGABRT, Aborted.                                /* 最开始要确定是什么信号量导致程序异常退出，这样很容易缩小范围，猜测大概可能发生了什么问题 */

#0  0xe9d2f630 in ?? ()

[Current thread is 1 (LWP 276)]

貌似找不到符号表，库文件匹配不上，怎么办？用set solib-search-path设置一下库文件路径就好了！

(gdb) set solib-search-path /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/mnt/work/linecard_app:/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/local:/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib:/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/local/libfhdrv_kdrv_board_impl.so...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/local/libvirtual_netdev_drv.so...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libpthread.so.0...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/librt.so.1...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/ld-linux.so.3...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libm.so.6...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libnss_files.so.2...(no debugging symbols found)...done.
/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/mnt/work/linecard_app

用bt命令看一下调用栈：

(gdb) bt
#0  0xe9d2f630 in raise () from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libc.so.6
#1  0xe9d309c8 in abort () from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libc.so.6
#2  0xf5a52158 in diag_displayTermPack (ucTermType=<optimized out>, usTermCmd=<optimized out>, pucData=0xc6e61440 "\001", usDataLen=<optimized out>)
    at ../../code/diag/diag/diag_display.c:147
#3  0xf5a62288 in wos_taskDiagMsgProc (pulPara=0xf5b20da8 <g_astWosPerfMsgDispatchStat+44>) at ../../code/diag/wos/wos_task.c:1973
#4  <signal handler called>         /* 触发信号量异常处理函数 */
#5  0xe9c71ca0 in __dynamic_cast () from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib/libstdc++.so.6
#6  0xf6416f70 in cfg_mod_set_onu_local_mng_interface_config_flag (pon_no=0, onu_no=0, cfg_flag=0 '\000')
    at /media/new/linyao/2_xPON/xPON/src/config_data_gpon/config_module_cinterface.cpp:35769
#7  0xf6cfa60c in str_ipaddr_to_general_ip (pgeneral_ip=0x1 <error: Cannot access memory at address 0x1>,
    str_ipaddr=0x80075 <error: Cannot access memory at address 0x80075>) at /media/new/linyao/2_xPON/xPON/src/voice/voice_service_global.cpp:945
#8  0xf5d382bc in CMscCommandObject::Parse (this=0x25a7a38, buf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0, volt_id=0 '\000', is_head=1 '\001')
    at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/msc_command_object.cpp:201
#9  0xf5d39d48 in CServiceObject::DispatchMscMessage (this=0x2598818, cmd_id=1045, pbuf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0,
    volt_id=0 '\000', is_head=1 '\001') at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/service_object.cpp:46
#10 0xf5d29f28 in CPonSystem::DispatchMscMessage (this=0x849a80, cmd_id=1045, pbuf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0, is_head=1 '\001',
    volt_id=0 '\000') at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/pon_system.hpp:566
#11 0xf5d25494 in pon_system_dispatch_msc (cmd_id=1045, pbuf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0, is_head=1 '\001')
    at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/pon_system_cinterface.cpp:385
#12 0xf728c46c in MSC_RegProcGswCmdFunToManager (cmdId=1045, cmdType=0 '\000', pFun=0xde83a22c, pPrmt=0x2a4 <error: Cannot access memory at address 0x2a4>)
    at /media/new/linyao/2_xPON/xPON/src/msc/msc_main.c:1477
#13 0xf72897dc in __gnu_cxx::__normal_iterator<CServiceObject**, std::vector<CServiceObject*, std::allocator<CServiceObject*> > >::__normal_iterator (this=0x0,
    __i=@0x1: <error reading variable>)
    at /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.24-binutils-2.25/usr/arm-buildroot-linux-gnueabi/include/c++/5.3.0/bits/stl_iterator.h:740
#14 0xf72897dc in __gnu_cxx::__normal_iterator<CServiceObject**, std::vector<CServiceObject*, std::allocator<CServiceObject*> > >::__normal_iterator (this=0xe3400002,
    __i=@0xe300c406: <error reading variable>)
    at /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.24-binutils-2.25/usr/arm-buildroot-linux-gnueabi/include/c++/5.3.0/bits/stl_iterator.h:740
#15 0xf72897dc in __gnu_cxx::__normal_iterator<CServiceObject**, std::vector<CServiceObject*, std::allocator<CServiceObject*> > >::__normal_iterator (this=0x0,
    __i=@0x0: <error reading variable>)
    at /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.24-binutils-2.25/usr/arm-buildroot-linux-gnueabi/include/c++/5.3.0/bits/stl_iterator.h:740
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

frame 4触发信号量异常处理函数，那么说明问题肯定出在frame 5。

(gdb) frame 5
#5  0xe9c71ca0 in __dynamic_cast () from /media/new/justin/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib/libstdc++.so.6
(gdb) info registers
r0             0xb8d70860 3101100128
r1             0xf5d6c6c0 4124493504
r2             0xf5d6c9e0 4124494304
r3             0x0 0
r4             0xb8d70860 3101100128
r5             0xf65f1808 4133427208
r6             0x79e0 31200
r7             0xf5af8ed4 4121923284
r8             0xf5af903c 4121923644
r9             0x64 100
r10            0xf5b1a118 4122059032
r11            0xc6e581ac 3336929708
r12            0x2564 9572
sp             0xc6e58120 0xc6e58120
lr             0xf6416f70 -163483792
pc             0xe9c71ca0 0xe9c71ca0 <__dynamic_cast+16>
cpsr           0xa0070010 -1610153968
(gdb) disassemble
Dump of assembler code for function __dynamic_cast:
   0xe9c71c90 <+0>: ldr r12, [r0]
   0xe9c71c94 <+4>: push {r4, r5, r6, r7, r8, lr}
   0xe9c71c98 <+8>: mov r4, r0
   0xe9c71c9c <+12>: sub sp, sp, #40 ; 0x28
=> 0xe9c71ca0 <+16>: ldr lr, [r12, #-8]                                            /* 出错指令，将r12-8指向内存地址的数据加载到lr，通过前面打印的寄存器值发现r12此时为0x2564，在0号page，明显的非法地址访问 */
(gdb) x/40wx $sp
0xc6e58120: 0xf5d60101 0xb8d6d50c 0xf5d6d1c4 0xf65f1808
0xc6e58130: 0xc6e5815c 0xf5d09c3c 0xc6e5815c 0x00000002
0xc6e58140: 0x0013849e 0x00080075 0xf6e05bb4 0xf65f1808
0xc6e58150: 0x000079e0 0xf5af8ed4 0xf5af903c 0xf6416f70
0xc6e58160: 0x00000000 0xf5ce3f5c 0x000079e0 0x00000415
0xc6e58170: 0x001381ac 0x00080075 0x023f3460 0x00000101
0xc6e58180: 0x023f3460 0x00080101 0x00083460 0x00000000
0xc6e58190: 0xb8d703b8 0xb8d70338 0xb8d4d4b0 0x8000007a
0xc6e581a0: 0xf6e05bb4 0x00000008 0xc6e60944 0xf6cfa60c
0xc6e581b0: 0xc6e60870 0x00000001 0x00000001 0x00000001

出错指令为ldr lr, [r12, #-8] 此时r12的值为0x2564，指向内存0页，为非法地址，因此该错误为非法地址访问。

上一行指令sub sp, sp, #40说明该函数的栈大小为40个字节，打印所有栈数据。

此时pc地址在libstdc++.so.6库的代码段__dynamic_cast函数 lr地址（函数返回地址）在libconfig_xxx.so的代码段，即调用ONU_CONFIGOBJECT()宏处。

(gdb) list *0xf6416f70
0xf6416f70 is in cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char) (/media/new/linyao/2_xPON/xPON/src/config_data_gpon/config_module_cinterface.cpp:35769).
35764         MAPLE_LOG(ULLOG_PRI_ERROR, "onuno %d\r\n", onu_no);
35765         return ERR_INVALID_ONU_NO;
35766     }
35767 
35768     /* »