Preface
 
    Sometimes we need to know the details of a program(porcess) when it is running even it's on the edge of crach.There's a lot of tools can be used to do these kind of job such as truss,strace,ltrace,pstack,gstack,gdb,etc.Truss and strace is focus on system call and signal generation whereas pstack,gstack,gdb are relevant with function call.Actually,pstack is a soft link of gstack,and gstack is a shell packaging script of gdb.
 
Introduce
 
    pt-pmp is also relies on linux gdb.It's a tool which aggregate gdb statck traces for a specified program especially for MySQL.pt-pmp can create and summarize full stack traces of linux process by which we can diagnose what the process is waiting for.It is mostly used in performance tuning scenarios.
 
Procedure
 
Usage
 pt-pmp [OPTIONS] [FILES]
Parameters(few parameters in this tool)
 --binary //Specify which binary program to trace(default 'mysqld').

 --pid //Specify a process id to trace.

 --iterations //Specify trace times(default "1").

 --interval //Number of seconds every time(default "0").

 --lines //Specify the number of functions to trace(default "0" means infinite).

 --save-samples //output aggregation into a file for later analysis.
Exapmle
 
Trace gdb stack of mysqld into a file(start mysqld and then execute pt-pmp).
 [root@zlm1 :: ~]

 #pt-pmp --save-samples pmp.log

 Sun Jul   :: CEST

 /usr/bin/pt-pmp: line : gdb: command not found //gdb package is not installed.

 [root@zlm1 :: ~]

 #yum install gdb //Install gdb package first.

 //Omitted

 Installed:

   gdb.x86_64 :7.6.-.el7                                                                                                                         

 Complete!

 [root@zlm1 :: ~]

 #pt-pmp --save-samples pmp.log

 Sun Jul   :: CEST

       __io_getevents_0_4(libaio.so.),LinuxAIOHandler::collect(os0file.cc:),LinuxAIOHandler::poll(os0file.cc:),os_aio_linux_handler(os0file.cc:),os_aio_handler(os0file.cc:),fil_aio_wait(fil0fil.cc:),io_handler_thread(srv0start.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_wait,wait(os0event.cc:),os_event::wait_low(os0event.cc:),srv_worker_thread(srv0srv.cc:),start_thread(libpthread.so.),clone(libc.so.)

        sigwait(libpthread.so.),signal_hand(mysqld.cc:),pfs_spawn_thread(pfs.cc:),start_thread(libpthread.so.),clone(libc.so.)

        sigwaitinfo(libc.so.),timer_notify_thread_func(posix_timers.c:),pfs_spawn_thread(pfs.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_wait,wait(os0event.cc:),os_event::wait_low(os0event.cc:),srv_purge_coordinator_suspend(srv0srv.cc:),srv_purge_coordinator_thread(srv0srv.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_wait,wait(os0event.cc:),os_event::wait_low(os0event.cc:),buf_resize_thread(buf0buf.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_wait,wait(os0event.cc:),os_event::wait_low(os0event.cc:),buf_dump_thread(buf0dump.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_wait,native_cond_wait(thr_cond.h:),my_cond_wait(thr_cond.h:),inline_mysql_cond_wait(thr_cond.h:),compress_gtid_table(thr_cond.h:),pfs_spawn_thread(pfs.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_timedwait,os_event::timed_wait(os0event.cc:),os_event::wait_time_low(os0event.cc:),srv_monitor_thread(srv0srv.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_timedwait,os_event::timed_wait(os0event.cc:),os_event::wait_time_low(os0event.cc:),srv_error_monitor_thread(srv0srv.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_timedwait,os_event::timed_wait(os0event.cc:),os_event::wait_time_low(os0event.cc:),pc_sleep_if_needed(buf0flu.cc:),buf_flush_page_cleaner_coordinator(buf0flu.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_timedwait,os_event::timed_wait(os0event.cc:),os_event::wait_time_low(os0event.cc:),lock_wait_timeout_thread(lock0wait.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_timedwait,os_event::timed_wait(os0event.cc:),os_event::wait_time_low(os0event.cc:),ib_wqueue_timedwait(ut0wqueue.cc:),fts_optimize_thread(fts0opt.cc:),start_thread(libpthread.so.),clone(libc.so.)

        pthread_cond_timedwait,os_event::timed_wait(os0event.cc:),os_event::wait_time_low(os0event.cc:),dict_stats_thread(dict0stats_bg.cc:),start_thread(libpthread.so.),clone(libc.so.)

        poll(libc.so.),vio_io_wait(viosocket.c:),vio_socket_io_wait(viosocket.c:),vio_read(viosocket.c:),net_read_raw_loop(net_serv.cc:),net_read_packet_header(net_serv.cc:),net_read_packet(net_serv.cc:),my_net_read(net_serv.cc:),Protocol_classic::read_packet(protocol_classic.cc:),Protocol_classic::get_command(protocol_classic.cc:),do_command(sql_parse.cc:),handle_connection(connection_handler_per_thread.cc:),pfs_spawn_thread(pfs.cc:),start_thread(libpthread.so.),clone(libc.so.)

        poll(libc.so.),Mysqld_socket_listener::listen_for_connection_event(socket_connection.cc:),connection_event_loop(connection_acceptor.h:),mysqld_main(connection_acceptor.h:),__libc_start_main(libc.so.),_start

        nanosleep(libpthread.so.),os_thread_sleep(os0thread.cc:),srv_master_sleep(srv0srv.cc:),srv_master_thread(srv0srv.cc:),start_thread(libpthread.so.),clone(libc.so.)

 //Above is the messages outupted on screen.All the contents is different functions relevant with linux and mysqld.

 //The output sorts with rules of most-frequen first.

 //The combination of function stacks lead by "__io_getevents_0_4(libaio.so.1)"(which is relevant with io) appears 10 times.
Check the content of output file.
#more pmp.log

TS 845703637.1530408066 -- ::

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ]

[New LWP ] //We got 28 new LWP from 3822 to 3981.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

0x00007f50efe74b7d in poll () from /lib64/libc.so.

Thread  (Thread 0x7f50e7ec1700 (LWP )): //It's the last thread in pmp.log file here.

#  0x00007f50efdbf7da in sigwaitinfo () from /lib64/libc.so.

#  0x0000000000f5171b in timer_notify_thread_func (arg=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./mysys/posix_timers.c:

#  0x0000000000f707b4 in pfs_spawn_thread (arg=0x30cb000) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/perfschema/pfs.cc:

#  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.

#  0x00007f50efe7f1ad in clone () from /lib64/libc.so.

Thread  (Thread 0x7f50ded07700 (LWP )):

#  0x00007f50f11b8644 in __io_getevents_0_4 () from /lib64/libaio.so.

#  0x000000000106a514 in LinuxAIOHandler::collect (this=0x7f50ded06e10) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  0x000000000106bc34 in LinuxAIOHandler::poll (this=0x7f50ded06e10, m1=0x7f50ded06ec8, m2=0x7f50ded06ec0, request=0x7f50ded06e90) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  0x000000000106d65c in os_aio_linux_handler (request=0x7f50ded06e90, m2=0x7f50ded06ec0, m1=0x7f50ded06ec8, global_segment=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  os_aio_handler (segment=, m1=0x7f50ded06ec8, m2=0x7f50ded06ec0, request=0x7f50ded06e90) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  0x000000000120196d in fil_aio_wait (segment=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/fil/fil0fil.cc:

#  0x000000000110e980 in io_handler_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/srv/srv0start.cc:

#  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.

#  0x00007f50efe7f1ad in clone () from /lib64/libc.so.

Thread  (Thread 0x7f50de506700 (LWP )):

#  0x00007f50f11b8644 in __io_getevents_0_4 () from /lib64/libaio.so.

#  0x000000000106a514 in LinuxAIOHandler::collect (this=0x7f50de505e10) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  0x000000000106bc34 in LinuxAIOHandler::poll (this=0x7f50de505e10, m1=0x7f50de505ec8, m2=0x7f50de505ec0, request=0x7f50de505e90) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  0x000000000106d65c in os_aio_linux_handler (request=0x7f50de505e90, m2=0x7f50de505ec0, m1=0x7f50de505ec8, global_segment=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  os_aio_handler (segment=, m1=0x7f50de505ec8, m2=0x7f50de505ec0, request=0x7f50de505e90) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/os/os0file.cc:

#  0x000000000120196d in fil_aio_wait (segment=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/fil/fil0fil.cc:

#  0x000000000110e980 in io_handler_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/innobase/srv/srv0start.cc:

#  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.

#  0x00007f50efe7f1ad in clone () from /lib64/libc.so.

//Omitted.

Thread  (Thread 0x7f50deeb2700 (LWP )):

#  0x00007f50f13c8ec1 in sigwait () from /lib64/libpthread.so.

#  0x00000000007c357b in signal_hand (arg=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/mysqld.cc:

#  0x0000000000f707b4 in pfs_spawn_thread (arg=0x386a740) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/perfschema/pfs.cc:

#  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.

#  0x00007f50efe7f1ad in clone () from /lib64/libc.so.

Thread  (Thread 0x7f50be7fc700 (LWP )):

#  0x00007f50f13c5705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.

#  0x0000000000ebe6f5 in native_cond_wait (mutex=<optimized out>, cond=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./include/thr_cond.h:

#  my_cond_wait (mp=<optimized out>, cond=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./include/thr_cond.h:

#  inline_mysql_cond_wait (src_line=, src_file=0x15d92e0 "/export/home/pb2/build/sb_0-26514852-1514433850.9/mysql-5.7.21/sql/rpl_gtid_persist.cc", mutex=<optimized out>, that=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./include/mysql/psi/mysql_thread.h:

#  compress_gtid_table (p_thd=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/rpl_gtid_persist.cc:

#  0x0000000000f707b4 in pfs_spawn_thread (arg=0x386a740) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/perfschema/pfs.cc:

#  0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.

#  0x00007f50efe7f1ad in clone () from /lib64/libc.so.

Thread  (Thread 0x7f50dee81700 (LWP )):

#  0x00007f50efe74b7d in poll () from /lib64/libc.so.

#  0x00000000012bae8f in vio_io_wait (vio=<optimized out>, event=<optimized out>, timeout=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./vio/viosocket.c:

#  0x00000000012baf83 in vio_socket_io_wait (vio=<optimized out>, event=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./vio/viosocket.c:

#  0x00000000012bb5b8 in vio_read (vio=0x7f50b0000bb0, buf=0x7f50b000a3c0 "\001", size=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./vio/viosocket.c:

#  0x0000000000c6de03 in net_read_raw_loop (net=0x7f50b0002570, count=) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/net_serv.cc:

#  0x0000000000c6e69b in net_read_packet_header (net=0x7f50b0002570) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/net_serv.cc:

#  net_read_packet (net=0x7f50b0002570, complen=0x7f50dee80d38) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/net_serv.cc:

#  0x0000000000c6e94c in my_net_read (net=0x7f50b0002570) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/net_serv.cc:

#  0x0000000000c7c1dc in Protocol_classic::read_packet (this=0x7f50b0001e18) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/protocol_classic.cc:

#  0x0000000000c7af52 in Protocol_classic::get_command (this=0x7f50b0001e18, com_data=0x7f50dee80de0, cmd=0x7f50dee80e0c) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/protocol_classic.cc:

# 0x0000000000d1cf47 in do_command (thd=0x7f50b0000dc0) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/sql_parse.cc:

# 0x0000000000ded7ac in handle_connection (arg=<optimized out>) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/conn_handler/connection_handler_per_thread.cc:

# 0x0000000000f707b4 in pfs_spawn_thread (arg=0x382fa60) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./storage/perfschema/pfs.cc:

# 0x00007f50f13c1df5 in start_thread () from /lib64/libpthread.so.

# 0x00007f50efe7f1ad in clone () from /lib64/libc.so.

Thread  (Thread 0x7f50f17e8740 (LWP )):

#  0x00007f50efe74b7d in poll () from /lib64/libc.so.

#  0x0000000000deeaf9 in Mysqld_socket_listener::listen_for_connection_event (this=0x30d96f0) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/conn_handler/socket_connection.cc:

#  0x00000000007c89c4 in connection_event_loop (this=0x373e050) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/conn_handler/connection_acceptor.h:

#  mysqld_main (argc=, argv=0x2f897b8) at /export/home/pb2/build/sb_0--1514433850.9/mysql-5.7./sql/mysqld.cc:

#  0x00007f50efdaaaf5 in __libc_start_main () from /lib64/libc.so.

#  0x00000000007be479 in _start ()

TS 788084395.1530408210 -- ::

[root@zlm1 :: ~]

#
Demonstrate of relevant conceptions.
 .What's gdb?

 gdb is tool of linux,a debugger of GNU project.It can help you to see what's going on inside the program while it executes.For example,the usual case is the program running slowly or almost to be crashed.

 .What can gdb do?

 It can catch details in the act belowl:

 ).Start the program and specify anything the might affect its behavior.

 ).Let the program stop on specific conditions.

 ).Check the details when the program stops.

 ).Change things in the program and try to correct effect of potential problems.

 .What's LWP?

 LWP is abbreviation of Light Weight Process.

 Process:You'll find a process in background after you've executed a linux command(which can be observed by "ps -ef").

 Thread:It's nothing but a flow of execution of the linux process.

 Light Weight Process:It's almost the same with "thread" but a term at kernal level.

 Here's some detail of it:

 What are Linux Processes, Threads, Light Weight Processes, and Process State

 https://www.thegeekstuff.com/2013/11/linux-process-and-threads/
Something about output file.
 .The second column in the file shows the detail of memory address with hex.

 .At the very beginning of the file in Thread (LWP ),it shows the detail of how MySQL process works.

 "Mysqld_socket_listener::listen_for_connection_event" is a MySQL function defined in source code file "socket_connection.cc".This is a clue which can help you to findout where to servay the details if something happened and stucked here.

 .At every bottom of each thread(except thread ),you can see the same contents below:

 "#. 0x... clone () from /lib64/libc.so.6" //clone() is a function to create a new thread.

 "#. 0x... start_thread () from /lib64/libpthread.so.0" //start_thread() is a function to start a poxis thread.
Summary
  • pt-pmp is read-only tool and it depends on the linux gdb to collect stack traces.
  • when pt-pmp works,it will attach gdb to the program you specified,what will probably freeze the program for a while.
  • It's not recommend to run the tool on a very busy product system because of the potential freezing period and resource consumption.
  • As a matter of fact,you can also do the stack tracing stuff by pstack tool but that's not today's motif.
 

Percona-Tookit工具包之pt-pmp的更多相关文章

  1. Linux后台开发工具箱

    https://files-cdn.cnblogs.com/files/aquester/Linux后台开发工具箱.pdf 目录 目录 1 1. 前言 3 2. 脚本类工具 3 2.1. sed命令- ...

  2. Mysql: pt-table-checksum 和 pt-table-sync 检查主从一致性，实验过程

    一.安装 percona 包 1.安装仓库的包 https://www.percona.com/doc/percona-repo-config/yum-repo.html sudo yum insta ...

  3. Linux后台开发工具箱-葵花宝典

    Linux后台开发工具箱-葵花宝典 一见 2016/11/4 目录 目录 1 1. 前言 4 2. 脚本类工具 4 2.1. 双引号和单引号 4 2.2. 取脚本完整文件路径 5 2.3. 环境变量和 ...

  4. 推荐几款MySQL相关工具

    前言: 随着互联网技术的不断发展, MySQL 相关生态也越来越完善,越来越多的工具涌现出来.一些公司或个人纷纷开源出一些不错的工具,本篇文章主要介绍几款 MySQL 相关实用工具.提醒下,这里并不介 ...

  5. [知识库分享系列] 二、.NET（ASP.NET）

    最近时间又有了新的想法,当我用新的眼光在整理一些很老的知识库时,发现很多东西都已经过时,或者是很基础很零碎的知识点.如果分享出去大家不看倒好,更担心的是会误人子弟,但为了保证此系列的完整,还是选择分享 ...

  6. SQL慢查询安装过程

    SQL慢查询 基本操作 打开防火墙 firewall-cmd --zone=public --add-port=3306/tcp --permanent firewall-cmd --reload 安 ...

  7. 快速安装Percona pt工具

    yum install perl-DBI perl-DBD-MySQL perl-Time-HiRes perl-Time-HiRes perl-IO-Socket-SSLwget http://pk ...

  8. Percona 工具包 pt-online-schema-change 简介

    mysql的在线表结构修改,因为低效和阻塞读写.一直被诟病.至于ALTER TABLE 的原理,参看我上一篇文章.MySQL在线修改大表结构.看完后,发现的问题是还是会锁的,且对于在线更新的这块也是不 ...

  9. 安装percona工具包

    1.安装percona源 sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona- ...

  10. percona pt toolkit 总结

    ##=====================================================##pt-osc之工作流程:1.检查更改表是否有主键或唯一索引,是否有触发器2.检查修改表 ...

随机推荐

  1. 服务间调用--feign跟ribbon

    微服务一般来说当然是多服务多实例的,那么这些服务之间如何相互调用呢?spring cloud之前我们用dubbo,把服务开放出来,在xml配好后就可以像调用本地service一样调用其它模块的服务了: ...

  2. GitKraken使用教程-基础部分(9)

    10.  合并分支并解决冲突(conflict) 1) 合并分支 在代码管理过程中,切换分支或者同步服务器代码时,常常会出现代码冲突的情况,这种情况出现的原因一般是由于两个分支对同一个文件进行修改, ...

  3. SVN仓库连同版本信息迁移新服务器的步骤

    SVN仓库连同版本信息迁移新服务器的步骤 步骤一:导出(1)链接原服务器,找到SVN Server安装路径下的bin文件,并复制文件路径,如 C:\Program File\SVN Server\bi ...

  4. 设计模式之装饰器模式io的小入门(十一)

    装饰器模式详解地址 原文总结 定义: 在不必改变原类文件和使用继承的情况下, 动态的扩展一个对象的功能. 通过创建一个包装对象, 也就是装饰来包裹真实的对象 部分详解提示 看了一些文档, 装饰器模式非 ...

  5. codevs 1213 解的个数

    1213 解的个数 时间限制: 1 s 空间限制: 128000 KB 题目等级 : 黄金 Gold       题目描述 Description 已知整数x,y满足如下面的条件: ax+by+c = ...

  6. CAD鼠标移动到对象时显示对象内容

    //定义事件         Editor ed = doc.Editor;         ed.PointMonitor += new PointMonitorEventHandler(ed_Po ...

  7. lrzsz的使用

    可以方便的在本地PC机和远程服务器之间传输文件. 1.下载 直接在centos上执行命令yum -y install lrzsz 2.上传文件 rz // 上传文件,执行命令rz,会跳出文件选择窗口, ...

  8. 用户表单事件(focus事件)

    以前做用户系统的时候经常用到表单验证,正则表达式事件来处理和绑定事件和进行事件,这里说的其实只是一小部分,也不是很值得写,但是今天遇到了还是写一下,毕竟基础还是蛮重要的,就算懂的童鞋,巩固一下也是好的 ...

  9. MySQL入门很简单: 15 java访问MySQL数据库

    1. 连接数据库 1.1 下载安装驱动 java通过JDBC(Java Database Connectivity,Java数据库连接)来访问MySQL数据库.JDBC的编程接口提供的接口和类与MyS ...

  10. win8下使用IIS服务器及自定义服务器端包含模块（SSI）步骤

    配置完过段时间就容易忘记,特此记录. 1.开启IIS服务器. 默认没有安装,需要先安装. 打开控制面板--> 打开“程序和功能”--> 左侧选择“启用或关闭windows功能”--> ...