PostgreSQL SystemTap on Linux

http://digoal126.wap.blog.163.com/w2/blogDetail.do;jsessionid=3949B03DE151DA0E55D807466C5E630B.yqblog35-8010?blogId=fks_087064087086083071087082094069072087087064092081081067080086&showRest=true&p=2&hostID=digoal@126

德哥@Digoal 2013-08-14 15:57

这个错误的原因是库文件版本不正确, 用来老版本, 使用eu-readelf查看stap文件的两个环境变量, 如下 : 

[root@db-172-16-3-39 ~]# eu-readelf -d /usr/bin/stap|grep -E "RPATH|RUNPATH"

  RPATH             Library rpath: [/usr/lib64/systemtap]

  RUNPATH           Library runpath: [/usr/lib64/systemtap]

将路径加入到LD_LIBRARY_PATH中.

[root@db-172-16-3-39 ~]# export LD_LIBRARY_PATH=/usr/lib64/systemtap:$LD_LIBRARY_PATH

stap现在正常了

[root@db-172-16-3-39 ~]# stap 

A script must be specified.

Systemtap translator/driver (version 1.8/0.152 non-git sources)

Copyright (C) 2005-2012 Red Hat, Inc. and others

This is free software; see the source for copying conditions.

enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS

Usage: stap [options] FILE         Run script in file.

   or: stap [options] -            Run script on stdin.

   or: stap [options] -e SCRIPT    Run given script.

   or: stap [options] -l PROBE     List matching probes.

   or: stap [options] -L PROBE     List matching probes and local variables.

测试 : 

[root@db-172-16-3-39 pg94]# vi tps.d 

probe begin

{

  printf("hello\n")

  exit()

}

[root@db-172-16-3-39 pg94]# stap tps.d 

Checking "/lib/modules/2.6.18-274.el5/build/.config" failed with error: No such file or directory

Incorrect version or missing kernel-devel package, use: yum install kernel-devel-2.6.18-274.el5.x86_64 

这个错误是由于未安装当前正在运行的kernel对应的kernel-devel包.

[root@db-172-16-3-39 pg94]# rpm -qa|grep kernel

kernel-headers-2.6.18-274.el5

kernel-xen-devel-2.6.18-274.el5

kernel-xen-2.6.18-274.el5

kernel-2.6.18-274.el5

[root@db-172-16-3-39 ~]# uname -a

Linux db-172-16-3-39.sky-mobi.com 2.6.18-274.el5 #1 SMP Fri Jul 22 04:43:29 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

 

安装对应的kernel-devel版本.

yum install kernel-devel-2.6.18-274.el5.x86_64

如果yum源中没有这个版本的kernel-devel包, 可以去安装光盘中找一找.

或者同时更新内核版本至新版本.

 yum install -y kernel.x86_64 kernel-devel.x86_64

Package kernel-2.6.18-348.12.1.el5.x86_64 already installed and latest version

Package kernel-devel-2.6.18-348.12.1.el5.x86_64 already installed and latest version

vi /boot/grub/grub.conf

default=0

timeout=5

splashimage=(hd0,0)/boot/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.18-348.12.1.el5)

        root (hd0,0)

        kernel /boot/vmlinuz-2.6.18-348.12.1.el5 ro root=LABEL=/ rhgb quiet

        initrd /boot/initrd-2.6.18-348.12.1.el5.img

重启服务器

现在stap工作正常了 : 

[root@db-172-16-3-39 postgresql-5e3e8e4]# stap -ve 'probe begin { log("hello world") exit() }'
Pass 1: parsed user script and 85 library script(s) using 146788virt/23676res/3000shr/21384data kb, in 170usr/0sys/173real ms.
Pass 2: analyzed script: 1 probe(s), 2 function(s), 0 embed(s), 0 global(s) using 147316virt/24396res/3224shr/21912data kb, in 10usr/0sys/6real ms.
Pass 3: using cached /root/.systemtap/cache/3b/stap_3b2eaec778ce9832b394535505dde575_838.c

Pass 4: using cached /root/.systemtap/cache/3b/stap_3b2eaec778ce9832b394535505dde575_838.ko
Pass 5: starting run.
hello world
Pass 5: run completed in 0usr/20sys/306real ms.

stap调试好后, 就可以用来跟踪postgresql了.

PostgreSQL编译时必须开启dtrace支持. 开启dtrace后, 数据库将启用代码中的探针或跟踪点.

PostgreSQL内建探针参考如下 : 

src/backend/utils/probes.d

http://www.postgresql.org/docs/devel/static/dynamic-trace.html

检查你的PostgreSQL是否开启了dtrace支持, 如下 : 

pg94@db-172-16-3-39-> pg_config --configure

'--prefix=/home/pg94/pgsql9.4devel' '--with-pgport=2999' '--with-perl' '--with-tcl' '--with-python' '--with-openssl' '--with-pam' '--without-ldap' '--with-libxml' '--with-libxslt' '--enable-thread-safety' '--with-wal-blocksize=16' '--enable-dtrace'

如果没有--enable-dtrace, 那么需要重新编译一下你的PostgreSQL软件.

 

stap测试脚本1.

[root@db-172-16-3-39 pg94]# cat postgresql-query.stp 

global query_time, query_summary

 

probe process("/home/pg94/pgsql9.4devel/bin/postgres").mark("query__start") {

  query_time[tid(), $arg1] = gettimeofday_us();

}

 

probe process("/home/pg94/pgsql9.4devel/bin/postgres").mark("query__done") {

  p = tid()

  t = query_time[p, $arg1]; delete query_time[p, $arg1]

  if (t) {

    query_summary[p] <<< (gettimeofday_us() - t);

  }

}

 

probe end {

  printf("\ntid count min(us) avg(us) max(us)\n");

  foreach (p in query_summary) {

    printf("%d %d %d %d %d\n", p, @count(query_summary[p]),

     @min(query_summary[p]), @avg(query_summary[p]), @max(query_summary[p]));

  }

}

执行stap : 

[root@db-172-16-3-39 pg94]# stap postgresql-query.stp 

执行以下SQL : 

[root@db-172-16-3-39 pg94]# stap postgresql-query.stp 

digoal=# begin;

BEGIN

digoal=# select * from test for update;

 id 

----

  1

  2

  3

  4

  5

  6

  7

  8

  9

 10

(10 rows)

digoal=# end;

COMMIT

digoal=# select txid_current();

 txid_current 

--------------

      5969062

(1 row)

结束stap, 输出 : 

[root@db-172-16-3-39 pg94]# stap postgresql-query.stp 

按键Ctrl+C, 输出 : 

tid count min(us) avg(us) max(us)

17112 4 46 3794 14885

这个tid对应PostgreSQL background process

[root@db-172-16-3-39 pg94]# ps -ewf|grep 17112

pg94     17112 17005  0 15:15 ?        00:00:00 postgres: postgres digoal [local] idle

digoal=# select pg_backend_pid();

 pg_backend_pid 

----------------

          17112

(1 row)

 

[小结]

1. 安装systemtap注意 : 

-- 需要安装kernel相关, 并且版本要一致, 启动的kernel版本也要一致 : 

 

kernel

kernel-debuginfo

kernel-devel

kernel-debuginfo的安装要用到debug源.

[root@db-172-16-3-39 pg94]# cd /etc/yum.repos.d/

[root@db-172-16-3-39 yum.repos.d]# ll

total 36

-rw-r--r-- 1 root root 1926 Aug 29  2011 CentOS-Base.repo

-rw-r--r-- 1 root root  631 Aug 29  2011 CentOS-Debuginfo.repo

-rw-r--r-- 1 root root  626 Aug 29  2011 CentOS-Media.repo

-rw-r--r-- 1 root root 5390 Aug 29  2011 CentOS-Vault.repo

[root@db-172-16-3-39 yum.repos.d]# cat CentOS-Debuginfo.repo 

# CentOS-Base.repo

#

# The mirror system uses the connecting IP address of the client and the

# update status of each mirror to pick mirrors that are updated to and

# geographically close to the client.  You should use this for CentOS updates

# unless you are manually picking other mirrors.

#

 

# All debug packages from all the various CentOS-5 releases

# are merged into a single repo, split by BaseArch

#

# Note: packages in the debuginfo repo are currently not signed

#

 

[debug]

name=CentOS-5 - Debuginfo

baseurl=http://debuginfo.centos.org/5/$basearch/

gpgcheck=0

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

enabled=0

这里用到的源名为debug.

yum --enablerepo=debug list kernel-debuginfo

yum --enablerepo=debug install kernel-debuginfo

安装细节参考此文 : 

http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaai.systemTap/liaaisystap_pdf.pdf

 

2. postgresql编译时必须加上--enable-dtrace参数, 否则stap时会出现类似以下错误.

[root@db-172-16-3-39 pg93]# stap test.stp 

semantic error: while resolving probe point: identifier 'process' at test.stp:3:7

        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lock__wait__start")

                      ^

 

semantic error: no match

semantic error: while resolving probe point: identifier 'process' at :8:7

        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lock__wait__done")

                      ^

 

semantic error: while resolving probe point: identifier 'process' at :17:7

        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lwlock__wait__start")

                      ^

 

semantic error: while resolving probe point: identifier 'process' at :22:7

        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lwlock__wait__done")

                      ^

 

semantic error: while resolving probe point: identifier 'process' at :32:7

        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lwlock__condacquire__fail")

                      ^

 

Pass 2: analysis failed.  Try again with another '--vp 01' option.

3. 允许stap需要root权限, 或者将用户加入stapdev或者stapsys, 以及stapusr组.

如果普通用户不加入这两个组, 执行stap将报错 : 

pg93@db-172-16-3-39-> stap -ve 'probe begin { log("hello world") exit() }'

You are trying to run systemtap as a normal user.

You should either be root, or be part of the group "stapusr" and possibly one of the groups "stapsys" or "stapdev".

Systemtap translator/driver (version 1.8/0.152 non-git sources)

加完组后正常

 

# usermod -G stapdev,stapusr pg94

pg94@db-172-16-3-39-> stap -ve 'probe begin { log("hello world") exit() }'

Pass 1: parsed user script and 85 library script(s) using 148892virt/23772res/3068shr/21396data kb, in 160usr/10sys/172real ms.

Pass 2: analyzed script: 1 probe(s), 2 function(s), 0 embed(s), 0 global(s) using 149420virt/24492res/3292shr/21924data kb, in 10usr/0sys/6real ms.

Pass 3: translated to C into "/tmp/stapcqtmUe/stap_758dbd41826239e5e3211a815f6bfc58_838_src.c" using 149420virt/24760res/3540shr/21924data kb, in 0usr/0sys/0real ms.

Pass 4: compiled C into "stap_758dbd41826239e5e3211a815f6bfc58_838.ko" in 910usr/110sys/1028real ms.

Pass 5: starting run.

hello world

Pass 5: run completed in 10usr/20sys/307real ms.

本文就介绍到这里, 下次将介绍如何使用postgresql的探针.

 

[参考]

1. http://pgfoundry.org/projects/dtrace/

2. http://www.fosslc.org/drupal/content/probing-postgresql-dtrace-and-systemtap

3. http://www.postgresql.org/docs/devel/static/dynamic-trace.html

4. http://www.postgresql.org/docs/devel/static/install-procedure.html

5. http://www.ppurl.com/2011/05/dtrace-dynamic-tracing-in-oracle-solaris-mac-os-x-and-freebsd.html

6. http://www.emm.usp.br/downloads/pg/PG_perf_bootcamp.pdf

7. https://wiki.postgresql.org/wiki/DTrace

8. http://www.ppurl.com/?s=systemtap

9. http://www.ibm.com/developerworks/cn/linux/l-systemtap/

10. http://sourceware.org/systemtap/wiki/HomePage

11. http://fruli.krunch.be/~krunch/systemtap-osdcfr-20101010.pdf

12. http://blog.endpoint.com/2009/05/postgresql-with-systemtap.html

13. https://www.evernote.com/shard/s48/sh/1ccb0466-79b7-4090-9a5d-9371358ac54d/b8434e3e3b3130ce72422b9ae067e7b9

14. http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaai.systemTap/liaaisystap_pdf.pdf

15. http://linux.chinaunix.net/techdoc/develop/2008/12/28/1055546.shtml

16. http://os.51cto.com/art/201305/395819.htm

17. https://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/?locale=en-US