内核调试神器SystemTap — 简介与使用（一）

a linux trace/probe tool.

官网：https://sourceware.org/systemtap/

简介

SystemTap是我目前所知的最强大的内核调试工具，有些家伙甚至说它无所不能：）

(1) 发展历程

Debuted in 2005 in Red Hat Enterprise Linux 4 Update 2 as a technology preview.

After four years in development, System 1.0 was released in 2009.

As of 2011 SystemTap runs fully supported in all Linux distributions.

(2) 官方介绍

SystemTap provides free software(GPL) infrastructure to simplify the gathering of information about the

running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates

the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot

sequence that may be otherwise required to collect data.

SystemTap provides a simple command line interface and scripting language for writing instrumentation for

a live running kernel plus user-space application. We are publishing samples, as well as enlarging the internal

"tapset" script library to aid reuse and abstraction.

Among other tracing/probing tools, SystemTap is the tool of choice for complex tasks that may require live analysis,

programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing

jobs.

Current project members include Red Hat, IBM, Hitachi, and Oracle.

(3) 获取源码

git clone git://sourceware.org/git/systemtap.git

安装

(1) Ubuntu发行版

1. 安装systemtap包

apt-get install systemtap

2. 安装依赖包

gcc：C语言编译器

elfutils：提供分析调试信息的库函数

linux-headers-generic：编译内核模块所需的内核头文件以及模块配置信息

3. 安装内核调试信息(kernel-debuginfo)

kernel-debuginfo提供了调试内核所需的符号表，如果没有安装的话SystemTap的威力就会大打折扣，

只能提供kprobes系列的功能。

下载地址：http://ddebs.ubuntu.com/pool/main/l/linux/

下载对应的内核版本，我的是linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb

下载后安装：dpkg -i linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb

4. 验证

stap -ve 'probe kernel.function("do_fork") { print("hello world\n") exit() }'

如果没有提示错误，就是安装成功了。

(2) CentOS/RedHat发行版

使用yum安装下列rpm包即可：

systemtap：SystemTap包

gcc：C语言编译器

elfutils：提供库函数来分析调试信息

kernel-devel：编译内核模块所需的内核头文件及模块配置信息

kernel-debuginfo：提供所需的内核调试信息来定位内核函数和变量的位置

使用

一些例子SystemTap的简单例子。

(1) stap

通常直接使用stap执行用SystemTap语法编写的脚本即可。

stap - systemtap script translator/driver

stap test.stp // .stp后缀的文件是用SystemTap语法编写的脚本

脚本主要元素：probe point + probe handler

stap [options] FILE // Run script in file

stap [options] -e SCRIPT // Run given script.

stap [options] -l PROBE // List matching probes.

stap [options] -L PROBE // List matching probes and local variables.

常用选项

-h：帮助

-g：guru模式，嵌入式C代码需要

-m：指定编译成的模块名称

-v：add verbosity to all passes

-k：不删除临时目录

-p NUM：stop after pass NUM 1-5, instead of 5 (parse, elaborate, translate, compile, run)

-b：bulk (percpu file) mode, 使用RelayFS将数据从内核空间传输到用户空间

-o FILE：输出到指定文件，而不是stdout

-c CMD：start the probes, run CMD, and exit when it finishes

stap是SystemTap的前端，当出现以下情况时退出：

1. The user interrupts the script with a CTRL-C.

2. The script executes the exit() function.

3. The script encounters a sufficient number of soft errors.

4. The monitored command started with the stap program's -c option exits.

(2) staprun

如果我们的输入不是.stp脚本，而是一个用stap生成的模块，那么就用staprun来执行。

staprun - systemtap runtime

staprun [OPTIONS] MODULE [MODULE-OPTIONS]

staprun的作用：

The staprun program is the back-end of the Systemtap tool. It expects a kernel module produced by

the front-end stap tool.

Splitting the systemtap tool into a front-end and a back-end allows a user to compile a systemtap script

on a development machine that has the kernel debugging information (need to compile the script) and

then transfer the resulting kernel module to a production machine that doesn't have any development

tools or kernel debugging information installed.

staprun is a part of the SystemTap package, dedicated to module loading and unloading and kernel-to-user

data transfer.

常用选项

-o FILE：Send output to FILE.

-D：Run in background. This requires '-o' option.

(3) 监测内核函数

一个简单脚本，每当内核函数do_fork()被调用时，显示调用它的进程名、进程ID、函数参数。

global proc_counter

probe begin {
	print("Started monitoring creation of new processes...Press ^C to terminate\n")
	printf("%-25s %-10s %-s\n", "Process Name", "Process ID", "Clone Flags")
}

probe kernel.function("do_fork") {
	proc_counter++
	printf("%-25s %-10d 0x%-x\n", execname(), pid(), $clone_flags)
}

probe end {
	printf("\n%d processes forked during the observed period\n", proc_counter)
}

(4) 监测系统调用

一个简单脚本，显示4秒内open系统调用的信息：调用进程名、进程ID、函数参数。

probe syscall.open
{
	printf("%s(%d) open(%s)\n", execname(), pid(), argstr)
}

probe timer.ms(4000) # after 4 seconds
{
	exit()
}

(5) 监测源文件中所有函数入口和出口

括号内的探测点描述包含三个部分：

function name part：函数名

@file name part：文件名

function line part：所在行号

例如：

probe kernel.function("*@net/socket.c") {}
probe kernel.function("*@net/socket.c").return {}

这里指定函数名为任意（用*表示），指定文件名为net/socket.c，探测函数的入口和返回。

还可以用“：行号”来指定行号。

(6) 查找匹配的内核函数和变量

查找名字中包含nit的内核函数：

stap -l 'kernel.function("*nit*")'

查找名字中包含nit的内核函数和变量：

stap -L 'kernel.function("*nit*")'

(7) 自带的用例集

/root/systemtap/testsuite/systemtap.examples/，包含了许多用例脚本。

主要有几个方面：

network、io、interrupt、locks、memory、process、virtualization等

(8) 监控所有进程的收发包情况

global recv, xmit

probe begin {
	printf("Starting network capture...Press ^C to terminate\n")
}

probe netdev.receive {
	recv[dev_name, pid(), execname()] <<< length
}

probe netdev.transmit {
	xmit[dev_name, pid(), execname()] <<< length
}

probe end {
	printf("\nCapture terminated\n\n")
	printf("%-5s %-15s %-10s %-10s %-10s\n",
		"If", "Process", "Pid", "RcvPktCnt", "XmtPktCnt")

	foreach([dev, pid, name] in recv) {
		recvcnt = @count(recv[dev, pid, name])
		xmtcnt =  @count(xmit[dev, pid, name])
		printf("%-5s %-15s %-10d %-10d %-10d\n", dev, name, pid, recvcnt, xmtcnt)
	}
}

(9) Systemtap usage stories and interesting demos

https://sourceware.org/systemtap/wiki/WarStories

官网提供的很多例子。