一个未完成的2.6.32-220内核踩内存crash分析记录

遇到一个crash，log如下：

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81166504>] s_show+0xe4/0x330
PGD  PUD 12666d8067 PMD
Oops:  [#] SMP
last sysfs file: /sys/devices/pci0000:/::03.0/::00.0/host0/port-:/expander-:/port-::/end_device-::/target0::/:::/block/sdw/stat
CPU
Modules linked in: **********************
Pid: , comm: slabtop Not tainted 2.6.-.el6.x86_64 # To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M.
RIP: :[<ffffffff81166504>]  [<ffffffff81166504>] s_show+0xe4/0x330
RSP: :ffff8817fc9e1d98  EFLAGS:
RAX: ffff880c2fc217c0 RBX: 00000000000003fb RCX: ffff880c2fc21800
RDX:  RSI:  RDI: ffff880c2fc217d0
RBP: ffff8817fc9e1e18 R08:  R09:
R10: ffffffff817a234e R11:  R12: 00000000000003fb
R13: ffffffff817a234e R14:  R15:
FS:  00007feb7bb6b700() GS:ffff8800283c0000() knlGS:
CS:   DS:  ES:  CR0:
CR2:  CR3: 000000125178a000 CR4: 00000000000006e0
DR0:  DR1:  DR2:
DR3:  DR6: 00000000ffff0ff0 DR7:
Process slabtop (pid: , threadinfo ffff8817fc9e0000, task ffff8817facd2100)
Stack:
 ffff8812666d8080 ffff881158954000 ffff881700000000 ffff880c2fc21800
<> ffff880c2fc217c0 ffff8817f3eee740 ffff880c2fd18498
<>  ffff880c2fd10440 ffff8817fc9e1e18 ffff8817f3eee740
Call Trace:
 [<ffffffff811a0a35>] seq_read+0xe5/0x3f0
 [<ffffffff811e35be>] proc_reg_read+0x7e/0xc0
 [<ffffffff8117ea75>] vfs_read+0xb5/0x1a0
 [<ffffffff810d68c2>] ? audit_syscall_entry+0xc2/0x2b0
 [<ffffffff8117ebb1>] sys_read+0x51/0x90
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
Code:    fe   4c 8b  c8  8b        c8 0f 1f  4d  ed  0f   4e   c7 c2 4e  7a  4d 0f  ea <> 8b  4d  c4   c3    fe  dd  8b    f0
RIP  [<ffffffff81166504>] s_show+0xe4/0x330
 RSP <ffff8817fc9e1d98>
CR2: 

堆栈如下：

crash> bt
PID:    TASK: ffff8817facd2100  CPU:   COMMAND: "slabtop"
bt: invalid kernel virtual address: 776f645f7570635f  type: "cpu_online_map"
 # [ffff8817fc9e1960] machine_kexec at ffffffff8103244b
 # [ffff8817fc9e19c0] crash_kexec at ffffffff810baf92
 # [ffff8817fc9e1a90] oops_end at ffffffff814fded0
 # [ffff8817fc9e1ac0] no_context at ffffffff810425db
 # [ffff8817fc9e1b10] __bad_area_nosemaphore at ffffffff81042865
 # [ffff8817fc9e1b60] bad_area at ffffffff8104298e
 # [ffff8817fc9e1b90] __do_page_fault at ffffffff810430c0
 # [ffff8817fc9e1cb0] do_page_fault at ffffffff814ffefe
 # [ffff8817fc9e1ce0] page_fault at ffffffff814fd255
    [exception RIP: s_show+]
    RIP: ffffffff81166504  RSP: ffff8817fc9e1d98  RFLAGS:
    RAX: ffff880c2fc217c0  RBX: 00000000000003fb  RCX: ffff880c2fc21800
    RDX:   RSI:   RDI: ffff880c2fc217d0
    RBP: ffff8817fc9e1e18   R8:    R9:
    R10: ffffffff817a234e  R11:   R12: 00000000000003fb
    R13: ffffffff817a234e  R14:   R15:
    ORIG_RAX: ffffffffffffffff  CS:   SS:
 # [ffff8817fc9e1e20] seq_read at ffffffff811a0a35
# [ffff8817fc9e1ea0] proc_reg_read at ffffffff811e35be
# [ffff8817fc9e1ef0] vfs_read at ffffffff8117ea75
# [ffff8817fc9e1f30] sys_read at ffffffff8117ebb1
# [ffff8817fc9e1f80] system_call_fastpath at ffffffff8100b0f2
    RIP: 000000370d0d83f0  RSP: 00007fff183a9450  RFLAGS:
    RAX:   RBX: ffffffff8100b0f2  RCX:
    RDX:   RSI: 00007feb7bb8a000  RDI:
    RBP: 000000000000079b   R8: 74616462616c7320   R9:
    R10:   R11:   R12:
    R13: 000000000000000a  R14: 000000000215a010  R15: 000000000000000a
    ORIG_RAX:   CS:   SS: 002b

函数端在s_show:

crash> dis -l s_show
dis: s_show: duplicate text symbols found:
ffffffff81023b70 (t) s_show /usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/arch/x86/kernel/cpu/mcheck/mce-severity.c:
ffffffff810b2d30 (t) s_show /usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/kernel/kallsyms.c:
ffffffff810f1800 (t) s_show /usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/kernel/trace/trace.c:
ffffffff8114e360 (t) s_show /usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/vmalloc.c:
ffffffff81166420 (t) s_show /usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c: 

发现很多个s_show的定义，所以反汇编下出错的地址：

[exception RIP: s_show+228]
RIP: ffffffff81166504

crash> dis -l ffffffff81166504
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166504 <s_show+>:        mov    (%rsi),%rsi

根据代码行，找到的函数是slab.c中的s_show,可以很明显根据堆栈看到最后回溯的rsi是空指针，所以会出现访问空指针的oops。

下面需要分析，rsi为啥是空指针。

crash> dis -l ffffffff81166420
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166420 <s_show>:    push   %rbp
0xffffffff81166421 <s_show+>:  mov    %rsp,%rbp
0xffffffff81166424 <s_show+>:  push   %r15
0xffffffff81166426 <s_show+>:  push   %r14
0xffffffff81166428 <s_show+>:  push   %r13
0xffffffff8116642a <s_show+>: push   %r12
0xffffffff8116642c <s_show+>: push   %rbx
0xffffffff8116642d <s_show+>: sub    $0x58,%rsp
0xffffffff81166431 <s_show+>: nopl   0x0(%rax,%rax,)
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166436 <s_show+>: mov    %rsi,%rax
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166439 <s_show+>: mov    %rdi,-0x58(%rbp)
0xffffffff8116643d <s_show+>: mov    %rsi,-0x50(%rbp)
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166441 <s_show+>: sub    $0x8058,%rax--------------------找到对应的4237行
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/include/linux/nodemask.h:
0xffffffff81166447 <s_show+>: mov    $0x200,%esi
0xffffffff8116644c <s_show+>: mov    $0xffffffff81c05280,%rdi
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166453 <s_show+>: mov    %rax,-0x38(%rbp)
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/include/linux/nodemask.h:
0xffffffff81166457 <s_show+>: callq  0xffffffff81275a10 <find_first_bit>
0xffffffff8116645c <s_show+>: cmp    $0x200,%eax
0xffffffff81166461 <s_show+>: mov    %eax,%edx
0xffffffff81166463 <s_show+>: mov    $0x200,%eax
0xffffffff81166468 <s_show+>: cmovg  %eax,%edx
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff8116646b <s_show+>: cmp    $0x1ff,%edx
0xffffffff81166471 <s_show+>: jg     0xffffffff81166730 <s_show+>
0xffffffff81166477 <s_show+>: xor    %r13d,%r13d
0xffffffff8116647a <s_show+>: movq   $0x0,-0x48(%rbp)
0xffffffff81166482 <s_show+>: movq   $0x0,-0x40(%rbp)
0xffffffff8116648a <s_show+>:        xor    %r15d,%r15d
0xffffffff8116648d <s_show+>:        xor    %ebx,%ebx
0xffffffff8116648f <s_show+>:        xor    %r12d,%r12d
0xffffffff81166492 <s_show+>:        nopw   0x0(%rax,%rax,)
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166498 <s_show+>:        mov    -0x38(%rbp),%rcx
0xffffffff8116649c <s_show+>:        movslq %edx,%rax
0xffffffff8116649f <s_show+>:        mov    0x8068(%rcx,%rax,),%rax
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff811664a7 <s_show+>:        test   %rax,%rax
0xffffffff811664aa <s_show+>:        je     0xffffffff811666f8 <s_show+>
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff811664b0 <s_show+>:        lea    0x40(%rax),%rcx
0xffffffff811664b4 <s_show+>:        mov    %rax,-0x60(%rbp)
0xffffffff811664b8 <s_show+>:        mov    %edx,-0x70(%rbp)
0xffffffff811664bb <s_show+>:        mov    %rcx,%rdi
0xffffffff811664be <s_show+>:        mov    %rcx,-0x68(%rbp)
0xffffffff811664c2 <s_show+>:        callq  0xffffffff814fcc50 <_spin_lock_irq>
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff811664c7 <s_show+>:        mov    -0x60(%rbp),%rax
0xffffffff811664cb <s_show+>:        mov    -0x70(%rbp),%edx
0xffffffff811664ce <s_show+>:        mov    -0x68(%rbp),%rcx
0xffffffff811664d2 <s_show+>:        mov    0x10(%rax),%rsi
0xffffffff811664d6 <s_show+>:        lea    0x10(%rax),%rdi
0xffffffff811664da <s_show+>:        cmp    %rdi,%rsi
0xffffffff811664dd <s_show+>:        je     0xffffffff81166513 <s_show+>
0xffffffff811664df <s_show+>:        mov    -0x38(%rbp),%r8
0xffffffff811664e3 <s_show+>:        mov    0x8018(%r8),%r9d
0xffffffff811664ea <s_show+>:        mov    %r9d,%r8d
0xffffffff811664ed <s_show+>:        nopl   (%rax)
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff811664f0 <s_show+>:        test   %r13,%r13
0xffffffff811664f3 <s_show+>:        jne    0xffffffff81166504 <s_show+>
0xffffffff811664f5 <s_show+>:        cmp    %r9d,0x20(%rsi)
0xffffffff811664f9 <s_show+>:        mov    $0xffffffff817a234e,%r10
0xffffffff81166500 <s_show+>:        cmovne %r10,%r13
/usr/src/debug/kernel-2.6.-.el6/linux-2.6.-.el6.x86_64/mm/slab.c:
0xffffffff81166504 <s_show+>:        mov    (%rsi),%rsi

根据代码行号4258行，可以确定 在访问 slabs_full链表时出错异常：

   static int s_show(struct seq_file *m, void *p)
    {
            struct kmem_cache *cachep = list_entry(p, struct kmem_cache, next);
            struct slab *slabp;
            unsigned long active_objs;
            unsigned long num_objs;
            unsigned long active_slabs = ;
            unsigned long num_slabs, free_objects = , shared_avail = ;
            const char *name;
            char *error = NULL;
            int node;
            struct kmem_list3 *l3;

            active_objs = ;
            num_slabs = ;
            for_each_online_node(node) {
                    l3 = cachep->nodelists[node];
                    if (!l3)
                            continue;

                    check_irq_on();
                    spin_lock_irq(&l3->list_lock);

                    list_for_each_entry(slabp, &l3->slabs_full, list) {

要想获取slabp，就得解析l3，要想解析l3，则需要解析cachep，要解析cachep，则需要解析传入的void*p，根据堆栈void*p是 seq_read中传入的。我们来看看这个*p到底是个什么参数：

根据反汇编代码，p就是一个头指针，它嵌入在kmem_cache中，

crash> struct -xo kmem_cache
struct kmem_cache {
     [0x0] struct array_cache *array[];
  [0x8000] unsigned int batchcount;
  [0x8004] unsigned int limit;
  [0x8008] unsigned int shared;
  [0x800c] unsigned int buffer_size;
  [0x8010] u32 reciprocal_buffer_size;
  [0x8014] unsigned int flags;
  [0x8018] unsigned int num;
  [0x801c] unsigned int gfporder;
  [0x8020] gfp_t gfpflags;
  [0x8028] size_t colour;
  [0x8030] unsigned int colour_off;
  [0x8038] struct kmem_cache *slabp_cache;
  [0x8040] unsigned int slab_size;
  [0x8044] unsigned int dflags;
  [0x8048] void (*ctor)(void *);
  [0x8050] const char *name;
  [0x8058] struct list_head next;------------嵌入

对函数反汇编：

crash> dis -l 0xffffffff81166420
/usr/src/debug/kernel-2.6.32-220.el6/linux-2.6.32-220.el6.x86_64/mm/slab.c: 4236
0xffffffff81166420 <s_show>:    push   %rbp
0xffffffff81166421 <s_show+1>:  mov    %rsp,%rbp
0xffffffff81166424 <s_show+4>:  push   %r15
0xffffffff81166426 <s_show+6>:  push   %r14
0xffffffff81166428 <s_show+8>:  push   %r13
0xffffffff8116642a <s_show+10>: push   %r12
0xffffffff8116642c <s_show+12>: push   %rbx
0xffffffff8116642d <s_show+13>: sub    $0x58,%rsp
0xffffffff81166431 <s_show+17>: nopl   0x0(%rax,%rax,1)
/usr/src/debug/kernel-2.6.32-220.el6/linux-2.6.32-220.el6.x86_64/mm/slab.c: 4237
0xffffffff81166436 <s_show+22>: mov    %rsi,%rax-------------------------------------rsi赋值给了rax，rsi中存放的是s_show函数的第二个参数*p
/usr/src/debug/kernel-2.6.32-220.el6/linux-2.6.32-220.el6.x86_64/mm/slab.c: 4236
0xffffffff81166439 <s_show+25>: mov    %rdi,-0x58(%rbp)
0xffffffff8116643d <s_show+29>: mov    %rsi,-0x50(%rbp)------------------------------rsi刚好又压栈了，所以根据rbp可以取出s_show的第二个参数*p
/usr/src/debug/kernel-2.6.32-220.el6/linux-2.6.32-220.el6.x86_64/mm/slab.c: 4237
0xffffffff81166441 <s_show+33>: sub    $0x8058,%rax
/usr/src/debug/kernel-2.6.32-220.el6/linux-2.6.32-220.el6.x86_64/include/linux/nodemask.h: 239
0xffffffff81166447 <s_show+39>: mov    $0x200,%esi
0xffffffff8116644c <s_show+44>: mov    $0xffffffff81c05280,%rdi
/usr/src/debug/kernel-2.6.32-220.el6/linux-2.6.32-220.el6.x86_64/mm/slab.c: 4237
0xffffffff81166453 <s_show+51>: mov    %rax,-0x38(%rbp)

查找rbp堆栈，然后-0x50，就可以获取到*p；

 #8 [ffff8817fc9e1ce0] page_fault at ffffffff814fd255
    [exception RIP: s_show+228]
    RIP: ffffffff81166504  RSP: ffff8817fc9e1d98  RFLAGS: 00010086
    RAX: ffff880c2fc217c0  RBX: 00000000000003fb  RCX: ffff880c2fc21800
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: ffff880c2fc217d0
    RBP: ffff8817fc9e1e18   R8: 0000000000000001   R9: 0000000000000001
    R10: ffffffff817a234e  R11: 0000000000000246  R12: 00000000000003fb
    R13: ffffffff817a234e  R14: 0000000000000400  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    ffff8817fc9e1ce8: 0000000000000000 0000000000000400
    ffff8817fc9e1cf8: ffffffff817a234e 00000000000003fb
    ffff8817fc9e1d08: ffff8817fc9e1e18 00000000000003fb
    ffff8817fc9e1d18: 0000000000000246 ffffffff817a234e
    ffff8817fc9e1d28: 0000000000000001 0000000000000001
    ffff8817fc9e1d38: ffff880c2fc217c0 ffff880c2fc21800
    ffff8817fc9e1d48: 0000000000000000 0000000000000000
    ffff8817fc9e1d58: ffff880c2fc217d0 ffffffffffffffff
    ffff8817fc9e1d68: ffffffff81166504 0000000000000010
    ffff8817fc9e1d78: 0000000000010086 ffff8817fc9e1d98
    ffff8817fc9e1d88: 0000000000000018 ffffffff811664c7
    ffff8817fc9e1d98: ffff8812666d8080 ffff881158954000
    ffff8817fc9e1da8: ffff881700000000 ffff880c2fc21800
    ffff8817fc9e1db8: ffff880c2fc217c0 ffff8817f3eee740
    ffff8817fc9e1dc8: ffff880c2fd18498 0000000000000000
    ffff8817fc9e1dd8: 0000000000000000 ffff880c2fd10440
    ffff8817fc9e1de8: ffff8817fc9e1e18 ffff8817f3eee740
    ffff8817fc9e1df8: ffff880c9ddbba80 ffff880c2fd18498
    ffff8817fc9e1e08: 0000000000000400 ffff8817fc9e1e60
    ffff8817fc9e1e18: ffff8817fc9e1e98 ffffffff811a0a35
 #9 [ffff8817fc9e1e20] seq_read at ffffffff811a0a35

crash> struct -xo kmem_cache
struct kmem_cache {
     [0x0] struct array_cache *array[4096];
  [0x8000] unsigned int batchcount;
  [0x8004] unsigned int limit;
  [0x8008] unsigned int shared;
  [0x800c] unsigned int buffer_size;
  [0x8010] u32 reciprocal_buffer_size;
  [0x8014] unsigned int flags;
  [0x8018] unsigned int num;
  [0x801c] unsigned int gfporder;
  [0x8020] gfp_t gfpflags;
  [0x8028] size_t colour;
  [0x8030] unsigned int colour_off;
  [0x8038] struct kmem_cache *slabp_cache;
  [0x8040] unsigned int slab_size;
  [0x8044] unsigned int dflags;
  [0x8048] void (*ctor)(void *);
  [0x8050] const char *name;
  [0x8058] struct list_head next;
  [0x8068] struct kmem_list3 *nodelists[512];
}
SIZE: 0x9068
crash> px 0xffff880c2fd18498-0x8058
$4 = 0xffff880c2fd10440
crash> struct kmem_cache 0xffff880c2fd10440
struct kmem_cache {
  array = {0xffff880c2fe93180, 0xffff880c1199cec0, 0xffff880c1199c6c0, 0xffff880c11a4cd80, 0xffff881811e1f580, 0xffff881811e1fd80, 0xffff881811e796c0, 0xffff881811e79ec0, 0xffff880c11a4c580, 0xffff880c11ae6cc0, 0xffff880c11ae64c0, 0xffff880c11b2bb80, 0xffff881811eee780, 0xffff881811f3a0c0, 0xffff881811f3a8c0, 0xffff881811f7d180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0...},
  batchcount = 12,
  limit = 24,
  shared = 8,
  buffer_size = 4096,
  reciprocal_buffer_size = 1048576,
  flags = 2147753984,
  num = 1,
  gfporder = 0,
  gfpflags = 0,
  colour = 0,
  colour_off = 64,
  slabp_cache = 0xffff880c2fc40100,---------------slab的管理数据和slab的obj分离，
  slab_size = 52,
  dflags = 0,
  ctor = 0x0,
  name = 0xffffffff817a24d8 "size-4096",
  next = {
    next = 0xffff880c2fd08458,
    prev = 0xffff880c2fd284d8
  },
  nodelists = {0xffff880c2fc217c0, 0xffff88182fc007c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0...}
}

根据找到的p值，我们确定了在遍历地址为0xffff880c2fd10440 的kmem_cache结构的nodelists的slabs_full 链表时，访问了空指针。

不过，因为nodelists是一个数组，我们现需要确定访问哪个下标时出错了。还好因为系统只有两个node，所以干脆从下标0遍历一下看：

crash> struct -xo kmem_list3
struct kmem_list3 {
   [0x0] struct list_head slabs_partial;
  [0x10] struct list_head slabs_full;
  [0x20] struct list_head slabs_free;
  [0x30] unsigned long free_objects;
  [0x38] unsigned int free_limit;
  [0x3c] unsigned int colour_next;
  [0x40] spinlock_t list_lock;
  [0x48] struct array_cache *shared;
  [0x50] struct array_cache **alien;
  [0x58] unsigned long next_reap;
  [0x60] int free_touched;
}
SIZE: 0x68

crash> rd 0xffff880c2fc217c0
ffff880c2fc217c0:  ffff880c2fc217c0 ffff880c2fc217c0   .../......./....
ffff880c2fc217d0:  ffff88049dd9ea80 ffff880c2fc23380   ........../....
ffff880c2fc217e0:  ffff880c2fc217e0 ffff880c2fc217e0   .../......./....
ffff880c2fc217f0:      ........a.......
ffff880c2fc21800:  0000000006d606d5 ffff880c2fe78800   .........../....
ffff880c2fc21810:  ffff880c2fc203e0 00000001030105e7   .../............
ffff880c2fc21820:      ................
ffff880c2fc21830:      ................
ffff880c2fc21840:  ffff880c2fc21840 ffff880c2fc21840   @../....@../....
ffff880c2fc21850:  ffff880c2fc21850 ffff880c2fc21850   P../....P../....
ffff880c2fc21860:  ffff880c2fc21860 ffff880c2fc21860   `../....`../....
ffff880c2fc21870:      ........a.......
ffff880c2fc21880:  00000000058f058f ffff880c2fe78400   .........../....
ffff880c2fc21890:  ffff880c2fc20400 00000001030105e7   .../............
ffff880c2fc218a0:      ................
ffff880c2fc218b0:      ................
ffff880c2fc218c0:  ffff880c2fc218c0 ffff880c2fc218c0   .../......./....
ffff880c2fc218d0:  ffff88079399db40 ffff880c2fc231c0   @........./....
ffff880c2fc218e0:  ffff8807989ff200 ffff88049dc62780   .........'......
ffff880c2fc218f0:      ........!.......
crash> slab ffff880c2fc217e0
struct slab {
  list = {
    next = 0xffff880c2fc217e0,
    prev = 0xffff880c2fc217e0
  },
  colouroff = ,
  s_mem = 0x61,
  inuse = ,
  free = ,
  nodeid =
}
crash> slab ffff880c2fc217c0
struct slab {
  list = {
    next = 0xffff880c2fc217c0,
    prev = 0xffff880c2fc217c0
  },
  colouroff = ,
  s_mem = 0xffff880c2fc23380,
  inuse = ,
  free = ,
  nodeid =
}

从上面的输出可以看出，nodelists [0]中对应的slabs_free链表为空，slabs_partial 链表为空， 只有 slabs_full 有数据。

slabs_full的地址就是nodelists[i] 的地址偏移0x10，遍历一下：

crash> list -s slab.inuse 0xffff88182fc007d0 >caq.slab_1
crash> list -s slab.inuse 0xffff880c2fc217d0 >caq.slab_0

由于list遇到null会认为结束，所以一开始list没出错，我还以为自己分析的地址有问题，打开我的输出文件才发现，确实slab的list出问题了。要知道，list访问到null或者访问到循环自己，

都会结束。

2035 ffff88049dd9af40
2036 inuse = 1
2037 ffff880799a9e2c0
2038 inuse = 0------------------full链表不可能inuse为0
2039 ffff880bee40a000
2040 inuse = 32768

查看一下内容：

crash> slab ffff880799a9e2c0
struct slab {
  list = {
    next = 0xffff880bee40a000,
    prev = 0x10000000000
  },
  colouroff = ,
  s_mem = 0xb00000100,
  inuse = ,
  free = ,
  nodeid =
}
crash> slab 0xffff880bee40a000
struct slab {
  list = {
    next = 0x0,-----------------------null指针出现了。
    prev = 0x2185b85600020
  },
  colouroff = ,
  s_mem = 0x2187025e00020,
  inuse = ,
  free = ,
  nodeid =
}

null指针出现了，该slab管理单元的prev已经不可信，所以要找到上一个slab，看ffff880799a9e2c0 ，发现它的数据有问题，才是导致这个oops的根本原因，因为ffff880799a9e2c0 中的内容

不是一个正常的slab，按照next访问的时候，才出现的异常，我们来看一下ffff880799a9e2c0  前后的内容。

ffff880799a9e200:   006e280a00000000   ....(........(n.
ffff880799a9e210:  000005240000230d 20f6cb8000000555   .#..$...U......
ffff880799a9e220:  002c000000002e1c 006e127600000000   ......,.....v.n.
ffff880799a9e230:      ................
ffff880799a9e240:   ffff88040000051d   ................
ffff880799a9e250:   ffff880b356be9c0   ..........k5....
ffff880799a9e260:  6664732f7665642f    /dev/sdf........
ffff880799a9e270:      ................
ffff880799a9e280:      ................
ffff880799a9e290:      ................
ffff880799a9e2a0:      ................
ffff880799a9e2b0:      ................
ffff880799a9e2c0:  ffff880bee40a000    ..@.............-------------------------------ffff880799a9e2c0地址的内容如下
ffff880799a9e2d0:  ffff88049dcd2000 0000000b00000100   . ..............
ffff880799a9e2e0:      ................
ffff880799a9e2f0:      ................
ffff880799a9e300:  00000000000f0015 30305f3661adaa00   ...........a6_00
ffff880799a9e310:  5f613030305f3031    10_000a_00000001
ffff880799a9e320:      .).... .........
ffff880799a9e330:      ................
ffff880799a9e340:   0000280a00000000   .............(..
ffff880799a9e350:      ................
ffff880799a9e360:   ffffffff81099c20   ........ .......
ffff880799a9e370:      ................

看一下ffff880799a9e2c0 本身属于什么数据：

kmem ffff880799a9e2c0
CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
ffff880c2fc40100 size-                                 4k
SLAB              MEMORY            TOTAL  ALLOCATED  FREE
ffff880799a9e000  ffff880799a9e140
FREE / [ALLOCATED]
  [ffff880799a9e2c0]

      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea001a99d290  799a9e000                            slab

由于kmem_cache的slab管理数据和slab的obj可以分离，所以根据 struct kmem_cache 0xffff880c2fd10440 对应的 slabp_cache  成员的值为 0xffff880c2fc40100 ，它也是一个kmem_cache

crash> struct -x kmem_cache 0xffff880c2fc40100
struct kmem_cache {
  array = {0xffff880c2fe9c000, 0xffff880c1199ec00, 0xffff880c11a3f000, 0xffff880c11a4f400, 0xffff881811e42000, 0xffff881811e85000, 0xffff881811ec1000, 0xffff881811ef4000, 0xffff880c11a99800, 0xffff880c11ae9c00, 0xffff880c11af9000, 0xffff880c11b2f400, 0xffff881811f26000, 0xffff881811f3f000, 0xffff881811f5d000, 0xffff881811f83000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0...},
  batchcount = 0x3c,
  limit = 0x78,
  shared = 0x8,
  buffer_size = 0x40,
  reciprocal_buffer_size = 0x4000000,
  flags = 0x42000,
  num = 0x3b,
  gfporder = 0x0,
  gfpflags = 0x0,
  colour = 0x0,
  colour_off = 0x40,
  slabp_cache = 0x0,
  slab_size = 0x140,
  dflags = 0x0,
  ctor = 0x0,
  name = 0xffffffff817a2435 "size-64",
  next = {
    next = 0xffff880c2fc38118,
    prev = 0xffff880c2fc58198
  },
  nodelists = {0xffff880c2fc21140, 0xffff88182fc00140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0...}
}

crash> struct kmem_list3 0xffff880c2fc217c0
struct kmem_list3 {
  slabs_partial = {
    next = 0xffff880c2fc217c0,
    prev = 0xffff880c2fc217c0
  },
  slabs_full = {
    next = 0xffff88049dd9ea80,
    prev = 0xffff880c2fc23380
  },
  slabs_free = {
    next = 0xffff880c2fc217e0,
    prev = 0xffff880c2fc217e0
  },
  free_objects = ,----------------free个数为0
  free_limit = ,
  colour_next = ,
  list_lock = {
    raw_lock = {
      slock =
    }
  },
  shared = 0xffff880c2fe78800,
  alien = 0xffff880c2fc203e0,
  next_reap = ,
  free_touched =
}

它是一个size-64的kmem_cache,也就是size 4096的cache的slab的管理数据，其实就是size-64的cache的obj。现在的问题是，这个obj被异常踩了，踩的地址是：ffff880799a9e2c0

我尝试找一下，跟这个内容一样的内存有没有：

crash> search ffff880bee40a000
ffff88049dd70d18: ffff880bee40a000
ffff880799a9e2c0: ffff880bee40a000
ffff880be23cd9c0: ffff880bee40a000

分别rd一下这三个地址，发现 ffff880799a9e2c0 和 ffff880be23cd9c0 中的内容是相同的：

crash> rd ffff880be23cd9c0
ffff880be23cd9c0:  ffff880bee40a000    ..@.............---------------------可能的源
ffff880be23cd9d0:  ffff88049dcd2000 0000000b00000100   . ..............
ffff880be23cd9e0:      ................
ffff880be23cd9f0:      ................
crash> rd ffff880799a9e2c0
ffff880799a9e2c0:  ffff880bee40a000    ..@.............----------------------被踩的，
ffff880799a9e2d0:  ffff88049dcd2000 0000000b00000100   . ..............
ffff880799a9e2e0:      ................
ffff880799a9e2f0:      ................

如上所示：两个地址里面的内容一模一样，

crash> kmem ffff880be23cd9c0
CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
ffff880c2fc00040 size-                                       4k
SLAB              MEMORY            TOTAL  ALLOCATED  FREE
ffff880be23cd000  ffff880be23cd200
FREE / [ALLOCATED]
  [ffff880be23cd9c0]

      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea002997d4d8  be23cd000                           slab

看一下这个slab的管理obj的数据的使用情况：

crash> slab ffff880be23cd000
struct slab {
  list = {
    next = 0xffff880c0b164000,
    prev = 0xffff880be6f95000
  },
  colouroff = ,
  s_mem = 0xffff880be23cd200,
  inuse = ,
  free = ,---------指向第一个free的节点，然后32节点中的数字指向下一个free的节点，
  nodeid =
}

crash> rd - 0xffff880c0b164030
ffff880c0b164030:        ............&...
ffff880c0b164040:   ffffffff 0000002e    #...............
ffff880c0b164050:  0000002e ffffffff ffffffff 0000001f   ................
ffff880c0b164060:  0000000d 0000002a 0000003f    ....*...?.......
ffff880c0b164070:   0000000f  0000002a   ............*...
ffff880c0b164080:    0000002c 0000003c   ...!...,...<...
ffff880c0b164090:  0000006b      k...........B...
ffff880c0b1640a0:  0000001b 0000001c 0000001d ffffffff   ................
ffff880c0b1640b0:  0000002a      *...............
ffff880c0b1640c0:     0000002b   #...$...'...+...
ffff880c0b1640d0:    0000003f 0000000b   )... ...?.......
ffff880c0b1640e0:   0000002c  0000000e   ....,...........
ffff880c0b1640f0:  0000002f ffffffff  0000002f   /.........../...
ffff880c0b164100:  0000003b  0000000b 0000003e   ;...........>...
ffff880c0b164110:    0000003b    ......;......
ffff880c0b164120:  0000003d 0000003e ffffffff    =...>.......&...
ffff880c0b164130:        ....@...A...B...
ffff880c0b164140:        C...D...E...F...
ffff880c0b164150:     0000004a   G...H...I...J...
ffff880c0b164160:  0000004b 0000004c 0000004d 0000004e   K...L...M...N...
ffff880c0b164170:  0000004f      O...P...Q...R...
ffff880c0b164180:        S...T...U...V...
ffff880c0b164190:     0000005a   W...X...Y...Z...
ffff880c0b1641a0:  0000005b 0000005c 0000005d 0000005e   [...\...]...^...
ffff880c0b1641b0:  0000005f      _...`...a...b...
ffff880c0b1641c0:        c...d...e...f...
ffff880c0b1641d0:     0000006a   g...h...i...j...
ffff880c0b1641e0:  0000006b 0000006c 0000006d ffffffff   k...l...m.......

根据上面的数据，也就是这个slab中目前空闲的为32-2a-3f-26-27-2b-b-1f(这个是最后一个，不能算free)，也就是7个free,根据slab重的105个in_use的统计，总共是112个，数据是ok的。

我们目前知道这个内存被踩了，但是这个地址不一定是被踩的初始地址，所以，有必要往上找，看哪个地址是被踩的初始地址（当然不排除踩多次）。

先尝试根据双向循环链表，恢复一下原来的链表。

ffff88049dd9af40的next指向了ffff880799a9e2c0，但由于 ffff880799a9e2c0 的地址里面数据是错的，所以 ffff880799a9e2c0 的数据不能用，但是 ffff880799a9e2c0 应该也是它的下一个

元素的prev指针，所以search一下，看谁的内存中有 ffff880799a9e2c0 这个值。

crash> search ffff880799a9e2c0
ffff8800390873c0: ffff880799a9e2c0
ffff88049dd9af40: ffff880799a9e2c0
ffff88079685a948: ffff880799a9e2c0
ffffea00102873c0: ffff880799a9e2c0
crash> kmem ffff8800390873c0
      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea0000c79d88                               reserved
crash> kmem ffff88079685a948
CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
ffff880c2fc40100 size-                                 4k
SLAB              MEMORY            TOTAL  ALLOCATED  FREE
ffff88079685a000  ffff88079685a140
FREE / [ALLOCATED]
  [ffff88079685a940]

      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea001a8ed3b0  79685a000                            slab
crash> kmem ffffea00102873c0
      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea0000c79d88                               reserved
crash> slab ffff88079685a940
struct slab {
  list = {
    next = 0xffff88079399a3c0,
    prev = 0xffff880799a9e2c0
  },
  colouroff = ,
  s_mem = 0xffff8806b875d000,
  inuse = ,
  free = ,
  nodeid =
}

查到了是 ffff88079685a940 这个prev是  0xffff880799a9e2c0，继续遍历940。

list  -s slab.inuse ffff88079685a940
。。。。

ffff880795767240
  inuse =
ffff880799a9e280
  inuse = 

最终遍历到240，又遇到一个异常的280，需要跟 0xffff880799a9e2c0类似，search找谁的内存中有280这个地址，最终恢复了原链表为：

f40->2c0->940->.....->240->280->ac0->....->f40，形成循环链表。

可以看出，280和2c0在地址上是连续的里面的内容全被踩了，那么踩内存的可能就是从280开始拷贝。

crash> rd ffff880be23cd900
ffff880be23cd900:      ................
ffff880be23cd910:      ................
ffff880be23cd920:      sdf.............
ffff880be23cd930:      ................
ffff880be23cd940:   ffff88040000051d   ................
ffff880be23cd950:   ffff880b356be9c0   ..........k5....
ffff880be23cd960:  6664732f7665642f    /dev/sdf........
ffff880be23cd970:      ................
ffff880be23cd980:      ................--------可能的源
ffff880be23cd990:      ................
ffff880be23cd9a0:      ................
ffff880be23cd9b0:      ................
ffff880be23cd9c0:  ffff880bee40a000    ..@.............
ffff880be23cd9d0:  ffff88049dcd2000 0000000b00000100   . ..............
ffff880be23cd9e0:      ................
ffff880be23cd9f0:      ................

crash> rd ffff880799a9e200
ffff880799a9e200:   006e280a00000000   ....(........(n.
ffff880799a9e210:  000005240000230d 20f6cb8000000555   .#..$...U......
ffff880799a9e220:  002c000000002e1c 006e127600000000   ......,.....v.n.
ffff880799a9e230:      ................
ffff880799a9e240:   ffff88040000051d   ................------------------已经被分配出去
ffff880799a9e250:   ffff880b356be9c0   ..........k5....
ffff880799a9e260:  6664732f7665642f    /dev/sdf........
ffff880799a9e270:      ................
ffff880799a9e280:      ................-------------目的地址
ffff880799a9e290:      ................
ffff880799a9e2a0:      ................
ffff880799a9e2b0:      ................
ffff880799a9e2c0:  ffff880bee40a000    ..@.............
ffff880799a9e2d0:  ffff88049dcd2000 0000000b00000100   . ..............
ffff880799a9e2e0:      ................
ffff880799a9e2f0:      ................
ffff880799a9e300:  00000000000f0015 30305f3661adaa00   ...........a6_00
ffff880799a9e310:  5f613030305f3031    10_000a_00000001

一种可能是我上面分析的，源地址这边memcpy，然后目的地址是我们的9e280，还有一种可能是，两者反过来，因为并不知道到底谁是源，甚至踩多次的情况，也就是拷贝多次。

感觉分析不下去了，这种踩内存不知道怎么分析。数据没有明显的特征,这个是同事遇到的，两次都没有帮到同事，心里很不是滋味，功力不够，继续修炼。