linux kernel elv_queue_empty野指针访问内核故障定位与解决

1. 故障描述

故障操作步骤：

单板上插了一个U盘，出问题前正在通过FTP往单板上拷贝文件，拷贝的过程中单板自动重启。

故障现象：

Entering kdb (current=0xc000000594069e38, pid 4) on processor 0 Oops: <NULL>

due to oops @ 0xffffffffc08d3d84

[0]more>

[0]kdb>

2. 信息采集

[0]kdb> bt

Stack traceback for pid 4

0xc000000594069e38 4 2 1 0 R 0xc00000059406a1a0 *ksoftirqd/0

Stack : fefc4d0e7b71a7af c00000058b4be318 c000000489904298 ffffffffc08d8174

0000000000000001 00000000f0000000 c000000489904258 ffffffffc0950f14

c000000594093c30 c000000594093c30 ffffffffc0e0f298 ffffffffc094ad00

c0000004b9780368 c00000058b5d6f30 c00000058b4be318 c00000058b4be318

fffffffffffffffb 0000000000000000 0000000000000000 c000000561912de8

c00000058b4be318 ffffffffc09534a0 c00000058b5d6f30 c000000561912de8

0000000000040000 ffffffffc09536ac 0000000000000000 c000000244074000

0000000000000000 ffffffffc0dddaa0 0000000000000101 ffffffffc0ddda80

ffffffffc0e71140 0000000000000000 ffffffffc0faf480 ffffffffc0f48c40

ffffffffc0de0000 ffffffffc08df110 c000000594093d20 c000000594093d20

...

Call Trace: [jiffies: 0x1003fe6ae]

[<ffffffffc08d3d84>] elv_queue_empty+0x24/0x48

[<ffffffffc08d7ee0>] __blk_run_queue+0x38/0x1d8

[<ffffffffc08d8174>] blk_run_queue+0x2c/0x50

[<ffffffffc0950f14>] scsi_run_queue+0x10c/0x418

[<ffffffffc09534a0>] scsi_next_command+0x48/0x68

[<ffffffffc09536ac>] scsi_io_completion+0x16c/0x550

[<ffffffffc08df110>] blk_done_softirq+0x98/0xb0

[<ffffffffc0679a58>] __do_softirq+0x120/0x1f0

[<ffffffffc0679ba0>] do_softirq+0x78/0x80

[<ffffffffc0679ca0>] ksoftirqd+0xf8/0x250

[<ffffffffc068fe9c>] kthread+0x94/0xa0

[<ffffffffc0638ef0>] kernel_thread_helper+0x10/0x20

<1>CPU 0 Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b6bab, epc == ffffffffc08d3d84, ra == ffffffffc08d7ee0

Cpu 0

$ 0 : 0000000000000000 0000000000000014 6b6b6b6b6b6b6b6b c00000058b4be318

$ 4 : c00000058b4be318 c0000004b83837e0 0000000000000000 c0000004b9780270

$ 8 : 0000000000000004 c0000004b97803b0 0000000000000001 0000000000275c43

$12 : 0000000000000028 ffffffffc0607568 ffffffffc0694b98 1ebdefda014b0000

$16 : c00000058b4be318 c000000489904298 c00000058b4be318 c0000004c6c59a38

$20 : c0000004c6c59a10 0400000000000000 ffffffffffffffbf 0000000000000001

$24 : 0000000000000004 ffffffffc06d7f50

$28 : c000000594090000 c000000594093bf0 ffffffffc0fc0000 ffffffffc08d7ee0

Hi : 0000000000000000

Lo : 0000000000000400

epc : ffffffffc08d3d84 elv_queue_empty+0x24/0x48

Not tainted

ra : ffffffffc08d7ee0 __blk_run_queue+0x38/0x1d8

Status: 5400ffe2 KX SX UX KERNEL EXL

Cause : 00800008

BadVA : 6b6b6b6b6b6b6bab

[0]kdb> md 0xc0000004b83837e0
0xc0000004b83837e0 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b83837f0 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b8383800 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b8383810 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b8383820 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b8383830 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b8383840 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6b kkkkkkkkkkkkkkkk
0xc0000004b8383850 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6ba5 kkkkkkkkkkkkkkk.

3. 故障分析

反汇编elv_queue_empty函数，计算得出异常指令为

“ffffffffc08d3e98: dca20000 ld v0,0(a1)

ffffffffc08d3e9c: dc590040 ld t9,64(v0)”

对于的C代码段为

“if (e->ops->elevator_queue_empty_fn)”

其中a1(e) = c0000004b83837e0, v0(e->ops) = 6b6b6b6b6b6b6b6b, 0x6b是free poisoning特征值，可以推断request_queue中的elevator字段指向的elevator_queue对象已经被kfree了，但是elevator字段还记录着其地址，导致e->ops为非法值。

从c0000004b83837e0 地址dump的数据特征看，共128字节被use-after-free poisoning 成0x6b，与struct elevator_queue大小一致，也佐证了上面的推断。

因为e对象已被释放，e此时已是野指针，e->ops为非法地址，所以e->ops->elevator_queue_empty_fn, 对非法地址进行寻址操作，从而导致异常。

int elv_queue_empty(struct request_queue *q) {

　　struct elevator_queue *e = q->elevator;

　　if (!list_empty(&q->queue_head))

　　　　return 0;

　　if (e->ops->elevator_queue_empty_fn)

　　　　return e->ops->elevator_queue_empty_fn(q);

　　return 1;

}

ffffffffc08d3e78 <elv_queue_empty>:

ffffffffc08d3e78: dc830000 ld v1,0(a0)

ffffffffc08d3e7c: dc850018 ld a1,24(a0)

ffffffffc08d3e80: 10830005 beq a0,v1,ffffffffc08d3e98 <elv_queue_empty+0x20>

ffffffffc08d3e84: 00000000 nop

ffffffffc08d3e88: 0000102d move v0,zero

ffffffffc08d3e8c: 03e00008 jr ra

ffffffffc08d3e90: 00000000 nop

ffffffffc08d3e94: 00000000 nop

ffffffffc08d3e98: dca20000 ld v0,0(a1)

ffffffffc08d3e9c: dc590040 ld t9,64(v0)

ffffffffc08d3ea0: 13200003 beqz t9,ffffffffc08d3eb0 <elv_queue_empty+0x38>

ffffffffc08d3ea4: 00000000 nop

ffffffffc08d3ea8: 03200008 jr t9

ffffffffc08d3eac: 00000000 nop

ffffffffc08d3eb0: 24020001 li v0,1

ffffffffc08d3eb4: 03e00008 jr ra

ffffffffc08d3eb8: 00000000 nop

ffffffffc08d3ebc: 00000000 nop

4. 解决方案

根据上述分析，得知异常是elevator_queue对象在销毁后，通过野指针访问时产生。

elevator_queue对象是在创建或销毁scsi_host设备对象是进行创建或销毁，其销毁流程的call trace如下：

scsi_host_dev_release -> scsi_free_queue -> blk_cleanup_queue -> elevator_exit -> elevator_release

反汇编scsi_io_completion函数，因为编译优化的原因，实际走的是scsi_end_request分支，即request queue中的request执行完成后的资源释放流程，和异常时打印的call trace稍有不同，

真实的异常时的call trace如下：

scsi_io_completion -> scsi_end_request -> scsi_next_command -> scsi_run_queue -> blk_run_queue -> __blk_run_queue -> _elv_queue_empty

综合上述分析，得出结论如下：

1 scsi host dev对象释放时，同时也释放了其关联的request_queue 和elevator_queue等对象，但是却没有及时更新request_queue对象中指向的elevator_queue对象的指针，导致此指针变成野指针；

2 对比2.6.32 LTS版本，发现scsi host dev对象释放时，其下挂的scsi_device对象也被释放，但是却没有及时更新其指向scsi_device对象的指针，即queuedata字段，导致此指针变成野指针。如果及时设置为空指针，scsi_run_queue就知道此scsi_device对象已经不存在，肯定也没有request queue，就可以直接返回，不再继续执行下去，避免后面的异常产生。

针对此两点结论，对以上2种野指针情况进行修改：

1 当释放elevator_queue对象后，及时将request_queue对象中指向的elevator_queue对象的指针设置为空；

2 当释放scsi_device对象后，及时将scsi host dev对象中指向的scsi_device对象的指针设置为空；

5. 参考资料

Make scsi_free_queue() kill pending SCSI commands

Linux Block Device Architecture