PostgreSQL SystemTap on Linux 转

PostgreSQL 支持动态跟踪, 可以通过dtrace或者systemtap工具统计相关探针的信息.
安装systemtap
yum install systemtap kernel-debuginfo kernel-devel
将安装以下包
systemtap-devel-1.8-.el5
systemtap-client-1.8-.el5
systemtap-runtime-1.8-.el5
后面将使用的stap命令
[root@db---- ~]# rpm -qf /usr/bin/stap
systemtap-devel-1.8-.el5
systemtap-client-1.8-.el5

检查stap是否正常
[root@db---- ~]# stap
stap: /usr/lib64/libelf.so.: version `ELFUTILS_1.' not found (required by stap)
stap: /usr/lib64/libdw.so.: version `ELFUTILS_0.' not found (required by stap)
stap: /usr/lib64/libdw.so.: version `ELFUTILS_0.' not found (required by stap)
stap: /usr/lib64/libdw.so.: version `ELFUTILS_0.' not found (required by stap)
stap: /usr/lib64/libdw.so.: version `ELFUTILS_0.' not found (required by stap)
stap: /usr/lib64/libdw.so.: version `ELFUTILS_0.' not found (required by stap)
这个错误的原因是库文件版本不正确, 用来老版本, 使用eu-readelf查看stap文件的两个环境变量, 如下 :
[root@db---- ~]# eu-readelf -d /usr/bin/stap|grep -E "RPATH|RUNPATH"
  RPATH             Library rpath: [/usr/lib64/systemtap]
  RUNPATH           Library runpath: [/usr/lib64/systemtap]
将路径加入到LD_LIBRARY_PATH中.
[root@db---- ~]# export LD_LIBRARY_PATH=/usr/lib64/systemtap:$LD_LIBRARY_PATH
stap现在正常了
[root@db---- ~]# stap
A script must be specified.
Systemtap translator/driver (version 1.8/0.152 non-git sources)
Copyright (C) - Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS
Usage: stap [options] FILE         Run script in file.
   or: stap [options] -            Run script on stdin.
   or: stap [options] -e SCRIPT    Run given script.
   or: stap [options] -l PROBE     List matching probes.
   or: stap [options] -L PROBE     List matching probes and local variables.
测试 :
[root@db---- pg94]# vi tps.d
probe begin
{
  printf("hello\n")
  exit()
}
[root@db---- pg94]# stap tps.d
Checking "/lib/modules/2.6.18-274.el5/build/.config" failed with error: No such file or directory
Incorrect version or missing kernel-devel package, use: yum install kernel-devel-2.6.-.el5.x86_64
这个错误是由于未安装当前正在运行的kernel对应的kernel-devel包.
[root@db---- pg94]# rpm -qa|grep kernel
kernel-headers-2.6.-.el5
kernel-xen-devel-2.6.-.el5
kernel-xen-2.6.-.el5
kernel-2.6.-.el5
[root@db---- ~]# uname -a
Linux db----.sky-mobi.com 2.6.-.el5 # SMP Fri Jul  :: EDT  x86_64 x86_64 x86_64 GNU/Linux

安装对应的kernel-devel版本.
yum install kernel-devel-2.6.-.el5.x86_64
如果yum源中没有这个版本的kernel-devel包, 可以去安装光盘中找一找.
或者同时更新内核版本至新版本.
 yum install -y kernel.x86_64 kernel-devel.x86_64
Package kernel-2.6.-348.12..el5.x86_64 already installed and latest version
Package kernel-devel-2.6.-348.12..el5.x86_64 already installed and latest version
vi /boot/grub/grub.conf
default=
timeout=
splashimage=(hd0,)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.-348.12..el5)
        root (hd0,)
        kernel /boot/vmlinuz-2.6.-348.12..el5 ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.-348.12..el5.img
重启服务器
现在stap工作正常了 :
[root@db---- postgresql-5e3e8e4]# stap -ve 'probe begin { log("hello world") exit() }'
Pass : parsed user script and  library script(s) using 146788virt/23676res/3000shr/21384data kb, in 170usr/0sys/173real ms.
Pass : analyzed script:  probe(s),  function(s),  embed(s),  global(s) using 147316virt/24396res/3224shr/21912data kb, in 10usr/0sys/6real ms.
Pass : using cached /root/.systemtap/cache/3b/stap_3b2eaec778ce9832b394535505dde575_838.c
Pass : using cached /root/.systemtap/cache/3b/stap_3b2eaec778ce9832b394535505dde575_838.ko
Pass : starting run.
hello world
Pass : run completed in 0usr/20sys/306real ms.
stap调试好后, 就可以用来跟踪postgresql了.
PostgreSQL编译时必须开启dtrace支持. 开启dtrace后, 数据库将启用代码中的探针或跟踪点.
PostgreSQL内建探针参考如下 :
src/backend/utils/probes.d
http://www.postgresql.org/docs/devel/static/dynamic-trace.html
检查你的PostgreSQL是否开启了dtrace支持, 如下 :
pg94@db-----> pg_config --configure
'--prefix=/home/pg94/pgsql9.4devel' '--with-pgport=2999' '--with-perl' '--with-tcl' '--with-python' '--with-openssl' '--with-pam' '--without-ldap' '--with-libxml' '--with-libxslt' '--enable-thread-safety' '--with-wal-blocksize=16' '--enable-dtrace'
如果没有--enable-dtrace, 那么需要重新编译一下你的PostgreSQL软件.

stap测试脚本1.
[root@db---- pg94]# cat postgresql-query.stp
global query_time, query_summary

probe process("/home/pg94/pgsql9.4devel/bin/postgres").mark("query__start") {
  query_time[tid(), $arg1] = gettimeofday_us();
}

probe process("/home/pg94/pgsql9.4devel/bin/postgres").mark("query__done") {
  p = tid()
  t = query_time[p, $arg1]; delete query_time[p, $arg1]
  if (t) {
    query_summary[p] <<< (gettimeofday_us() - t);
  }
}

probe end {
  printf("\ntid count min(us) avg(us) max(us)\n");
  foreach (p in query_summary) {
    printf("%d %d %d %d %d\n", p, @count(query_summary[p]),
     @min(query_summary[p]), @avg(query_summary[p]), @max(query_summary[p]));
  }
}
执行stap :
[root@db---- pg94]# stap postgresql-query.stp
执行以下SQL :
[root@db---- pg94]# stap postgresql-query.stp
digoal=# begin;
BEGIN
digoal=# select * from test for update;
 id
----

( rows)
digoal=# end;
COMMIT
digoal=# select txid_current();
 txid_current
--------------

( row)
结束stap, 输出 :
[root@db---- pg94]# stap postgresql-query.stp
按键Ctrl+C, 输出 :
tid count min(us) avg(us) max(us)

这个tid对应PostgreSQL background process
[root@db---- pg94]# ps -ewf|grep
pg94         : ?        :: postgres: postgres digoal [local] idle
digoal=# select pg_backend_pid();
 pg_backend_pid
----------------

( row)

[小结]
. 安装systemtap注意 :
-- 需要安装kernel相关, 并且版本要一致, 启动的kernel版本也要一致 :
kernel
kernel-debuginfo
kernel-devel
kernel-debuginfo的安装要用到debug源.
[root@db---- pg94]# cd /etc/yum.repos.d/
[root@db---- yum.repos.d]# ll
total
-rw-r--r--  root root  Aug    CentOS-Base.repo
-rw-r--r--  root root   Aug    CentOS-Debuginfo.repo
-rw-r--r--  root root   Aug    CentOS-Media.repo
-rw-r--r--  root root  Aug    CentOS-Vault.repo
[root@db---- yum.repos.d]# cat CentOS-Debuginfo.repo
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#

# All debug packages from all the various CentOS- releases
# are merged into a single repo, split by BaseArch
#
# Note: packages in the debuginfo repo are currently not signed
#

[debug]
name=CentOS- - Debuginfo
baseurl=http://debuginfo.centos.org/5/$basearch/
gpgcheck=
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
enabled=
这里用到的源名为debug.
yum --enablerepo=debug list kernel-debuginfo
yum --enablerepo=debug install kernel-debuginfo
安装细节参考此文 :
http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaai.systemTap/liaaisystap_pdf.pdf

. postgresql编译时必须加上--enable-dtrace参数, 否则stap时会出现类似以下错误.
[root@db---- pg93]# stap test.stp
semantic error: while resolving probe point: identifier 'process' at test.stp::
        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lock__wait__start")
                      ^

semantic error: no match
semantic error: while resolving probe point: identifier 'process' at ::
        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lock__wait__done")
                      ^

semantic error: while resolving probe point: identifier 'process' at ::
        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lwlock__wait__start")
                      ^

semantic error: while resolving probe point: identifier 'process' at ::
        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lwlock__wait__done")
                      ^

semantic error: while resolving probe point: identifier 'process' at ::
        source: probe process("/opt/pgsql9.3beta2/bin/postgres").mark("lwlock__condacquire__fail")
                      ^

Pass : analysis failed.  Try again with another '--vp 01' option.
. 允许stap需要root权限, 或者将用户加入stapdev或者stapsys, 以及stapusr组.
如果普通用户不加入这两个组, 执行stap将报错 :
pg93@db-----> stap -ve 'probe begin { log("hello world") exit() }'
You are trying to run systemtap as a normal user.
You should either be root, or be part of the group "stapusr" and possibly one of the groups "stapsys" or "stapdev".
Systemtap translator/driver (version 1.8/0.152 non-git sources)
加完组后正常
# usermod -G stapdev,stapusr pg94
pg94@db-----> stap -ve 'probe begin { log("hello world") exit() }'
Pass : parsed user script and  library script(s) using 148892virt/23772res/3068shr/21396data kb, in 160usr/10sys/172real ms.
Pass : analyzed script:  probe(s),  function(s),  embed(s),  global(s) using 149420virt/24492res/3292shr/21924data kb, in 10usr/0sys/6real ms.
Pass : translated to C into "/tmp/stapcqtmUe/stap_758dbd41826239e5e3211a815f6bfc58_838_src.c" using 149420virt/24760res/3540shr/21924data kb, in 0usr/0sys/0real ms.
Pass : compiled C into "stap_758dbd41826239e5e3211a815f6bfc58_838.ko" in 910usr/110sys/1028real ms.
Pass : starting run.
hello world

http://blog.163.com/digoal@126/blog/static/16387704020137140265557/