Percona-Tookit工具包之pt-heartbeat

 

Preface

 

    Replication delay is a common issue in MySQL replications.Especially in those replications not using semi-synchronous method.Long term delay leads to inconsistency between master and slaves.Therefore,we should regard the replication delay as an important monitor item.To a certain extent,the value of "Seconds_Behind_Master" shows the situation of replication delay,but it's always inaccurate.Even if the value is "0" sometimes,the replication delay remains.

 

Introduce

 

    pt-heartbeat is the very tool can be used to monitor replication delay of MySQL or PostgreSQL.It provides a way to update a master or to monitor a replica.If you do not speicify the connection options,it will retrieve them in your default my.cnf file.

 

Procedure

 

Usage

 pt-heartbeat [OPTIONS] [DSN] --update|--monitor|--check|--stop

Common parameters

 --check //Specify the check mode to slave delay.It only checks once then exits.
 --monitor //Specify the monitor mode to slave delay.
 --update //Specify the update mode on master.
 --check-read-only //It skips insert operations if the server is read-only status.
 --create-table //Create a heartbeat table.
 --table //Specify the heartbeat table(default "heartbeat").
 --daemonize //Run it as background process.
 --dbi-driver //Specify the drive of database(default "mysql",other value is "pg").
 --file //Specify a output file when using --monitor.
 --log //Print all output to the file when using --daemonize
 --master-server-id //Calculate delay from this master server id.
 --interval //Specify the frequency to update or check the heartbeat table(default "1.0").
 --run-time //Specify the period of time to run(default suffix is "s",if you set 60,it means 60s).

 Notice:
 Specify at least one of --stop, --update, --monitor, or --check.
 --update, --monitor, and --check are mutually exclusive.
 --daemonize and --check are mutually exclusive.

Examples

 

Execute pt-heartbeat to update the heartbeat of master.

 [root@zlm2 :: ~]
 #pt-heartbeat --database=sysbench --create-table --table=hb --daemonize -h192.168.1. -P3306 -uzlm -pzlmzlm --update

 [root@zlm2 :: ~]
 #

 (zlm@192.168.1.101 )[sysbench]>show tables like 'hb';
 +-------------------------+
 | Tables_in_sysbench (hb) |
 +-------------------------+
 | hb                      |
 +-------------------------+
  row in set (0.00 sec)

 (zlm@192.168.1.101 )[sysbench]>select * from hb;
 +----------------------------+-----------+------------------+----------+-----------------------+---------------------+
 | ts                         | server_id | file             | position | relay_master_log_file | exec_master_log_pos |
 +----------------------------+-----------+------------------+----------+-----------------------+---------------------+
 | --19T03::07.005490 |    | mysql-bin. |    | NULL                  |                NULL |
 +----------------------------+-----------+------------------+----------+-----------------------+---------------------+
  row in set (0.00 sec)

 //The operation above will create a heartbeat table "hb" in database "sysbench" if the target table not exist.
 //It will update the table which contains the heartbeat info of master every time you execute with "--update".

Execute pt-heartbeat to check delay of slave.

 [root@zlm2 :: ~]
 #pt-heartbeat --database=sysbench -h192.168.1. -P3306 -uzlm -pzlmzlm --check
 DBD::mysql::db selectrow_arrayref failed: Table 'sysbench.heartbeat' doesn't exist [for Statement "SHOW CREATE TABLE `sysbench`.`heartbeat`"] at /usr/bin/pt-heartbeat line 5873.

 [root@zlm2 :: ~]
 #pt-heartbeat --database=sysbench --create-table --table=hb -h192.168.1. -P3306 -uzlm -pzlmzlm --check
 0.00

 (zlm@192.168.1.102 )[sysbench]>show tables like 'hb';
 +-------------------------+
 | Tables_in_sysbench (hb) |
 +-------------------------+
 | hb                      |
 +-------------------------+
  row in set (0.00 sec)

 (zlm@192.168.1.102 )[sysbench]>select * from hb;
 +----------------------------+-----------+------------------+----------+-----------------------+---------------------+
 | ts                         | server_id | file             | position | relay_master_log_file | exec_master_log_pos |
 +----------------------------+-----------+------------------+----------+-----------------------+---------------------+
 | --19T03::53.005890 |    | mysql-bin. |   | NULL                  |                NULL |
 | -- ::        |    | NULL             |     NULL | NULL                  |                NULL |
 +----------------------------+-----------+------------------+----------+-----------------------+---------------------+
  rows in set (0.00 sec)

 //The connection option refers to slave.
 //It will also create the same heartbeat table on the slave.
 //The output will show the delay time.

Execute pt-heartbeat to monitor delay of slave.

 [root@zlm2 :: ~]
 #pt-heartbeat --database=sysbench --create-table --table=hb --daemonize --log=/root/monitor.log -h192.168.1. -P3306 -uzlm -pzlmzlm --monitor

 [root@zlm2 :: ~]
 #ls -l|grep monitor
 -rw-r--r--    root root     Jul  : monitor.log

 [root@zlm2 :: ~]
 #cat monitor.log
 *******************************************************************
  Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
  is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
  possibly with SSL_ca_file|SSL_ca_path for verification.
  If you really don't want to verify the certificate and keep the
  connection open to Man-In-The-Middle attacks please set
  SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
 *******************************************************************
   at /usr/bin/pt-heartbeat line .
 *******************************************************************
  Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
  is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
  possibly with SSL_ca_file|SSL_ca_path for verification.
  If you really don't want to verify the certificate and keep the
  connection open to Man-In-The-Middle attacks please set
  SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
 *******************************************************************
   at /usr/bin/pt-heartbeat line .

 # A software update is available:
 #   * The current version for Percona::Toolkit is 3.0.

 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]
 .00s [  .00s,  .00s,  .00s ]

 [root@zlm2 :: ~]
 #

 //The output will be write into the logfile sepcified by option "--log".
 //The logfile will be continuously written unless you've specified th option "--run-time".
 //It seems no delay between master and slave at all.We can use sysbench generate some transactions to observe the difference.

Execute sysbench and watch again.

 [root@zlm2 :: ~/sysbench-1.0/src/lua]
 #sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port= --mysql-user=zlm --mysql-password=zlmzlm --mysql-db=sysbench --tables= --table-size= --mysql-storage-engine=innodb prepare
 sysbench 1.0. (using bundled LuaJIT 2.1.-beta2)

 Creating table 'sbtest1'...
 Inserting  records into 'sbtest1'
 FATAL: mysql_drv_query() returned error  (The table 'sbtest1' is full) for query 'INSERT INTO sbtest1(k, c, pad) VALUES(5007652, '---------', '----'),(4980838, '---------', '----'),(4369690, '---------', '----'),(4980019, '---------', '----'),(4997783, '---------', '----'),(4996542, '---------', '----'),(5016589, '---------', '----'),(4994933, '---------', '----'),(5024203, '---------', '----'),(4162450, '---------', '----'),(4995444, '---------', '----'),(5014023, '---------', '----'),(5024261, '---------', '----'),(5024236, '---------', '----'),(5049980, '---------', '----'),(5471904, '---------', '----'),(5019788, '---------', '----'),(5021303, '---------', '----'),(5015123, '---------', '----'),(3676715, '---------', '----'),(5236763, '----FATAL: `sysbench.cmdline.call_command' function failed: ./oltp_common.lua:230: db_bulk_insert_next() failed

 //Master
 [root@zlm2 :: ~/sysbench-1.0/src/lua]
 #df -h
 Filesystem               Size  Used Avail Use% Mounted on
 /dev/mapper/centos-root  .4G  .4G   68M % / //The disk space of master is full.
 devtmpfs                 488M       488M   % /dev
 tmpfs                    497M       497M   % /dev/shm
 tmpfs                    497M  6.6M  491M   % /run
 tmpfs                    497M       497M   % /sys/fs/cgroup
 /dev/sda1                497M  118M  379M  % /boot
 none                      87G   80G  .6G  % /vagrant

 //Slave
 [root@zlm3 :: ~]
 #df -h
 Filesystem               Size  Used Avail Use% Mounted on
 /dev/mapper/centos-root  .4G  .4G   20K % / //The disk space of slave is full,too.
 devtmpfs                 488M       488M   % /dev
 tmpfs                    497M       497M   % /dev/shm
 tmpfs                    497M  6.5M  491M   % /run
 tmpfs                    497M       497M   % /sys/fs/cgroup
 /dev/sda1                497M  118M  379M  % /boot
 none                      87G   80G  .6G  % /vagrant

 //Check the output logfile.
 [root@zlm2 :: ~]
 #tail monitor.log
 .00s [  .40s,  .65s,  .55s ]
 .00s [  .26s,  .65s,  .55s ]
 .99s [  .13s,  .65s,  .55s ]
 .99s [  .00s,  .66s,  .55s ]
 .99s [  .86s,  .67s,  .56s ]
 .99s [  .73s,  .68s,  .56s ]
 .99s [  .59s,  .70s,  .57s ]
 .00s [  .46s,  .72s,  .57s ]
 .01s [  .29s,  .73s,  .58s ]
 .99s [  .16s,  .75s,  .58s ]

 //Long time later.

 [root@zlm2 :: ~]
 #tail monitor.log
 .00s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]

 [root@zlm2 :: ~]
 #tail monitor.log
 .00s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]
 .99s [ .50s, .50s, .50s ]
 .00s [ .50s, .50s, .50s ]

 (zlm@192.168.1.102 )[sysbench]>show slave status\G
 ERROR  (HY000): Lost connection to MySQL server during query
 (zlm@192.168.1.102 )[sysbench]>show slave status\G
 ERROR  (HY000): MySQL server has gone away
 No connection. Trying to reconnect...
 Connection id:
 Current database: sysbench

 //The slave has hung because of the lack of disk space.
 //The delay continuously increasing what can be seen in the logfile on master.

 #  :: server id   end_log_pos          Table_map: `sysbench`.`hb` mapped to number
 # at
 #  :: server id   end_log_pos          Update_rows: table id  flags: STMT_END_F
 ### UPDATE `sysbench`.`hb`
 ### WHERE
 ###   @='2018-07-19T04:31:55.004000' /* VARSTRING(78) meta=78 nullable=0 is_null=0 */
 ###   @= /* INT meta=0 nullable=0 is_null=0 */
 ###   @='mysql-bin.000019' /* VARSTRING(765) meta=765 nullable=1 is_null=0 */
 ###   @= /* LONGINT meta=0 nullable=1 is_null=0 */
 ###   @=NULL /* VARSTRING(765) meta=765 nullable=1 is_null=1 */
 ###   @=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
 ### SET
 ###   @='2018-07-19T04:31:55.004400' /* VARSTRING(78) meta=78 nullable=0 is_null=0 */
 ###   @= /* INT meta=0 nullable=0 is_null=0 */
 ###   @='mysql-bin.000019' /* VARSTRING(765) meta=765 nullable=1 is_null=0 */
 ###   @= /* LONGINT meta=0 nullable=1 is_null=0 */
 ###   @=NULL /* VARSTRING(765) meta=765 nullable=1 is_null=1 */
 ###   @=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
 # at
 #  :: server id   end_log_pos          Xid =
 COMMIT/*!*/;

 //The output above is the detail of updating heartbeat table "hb" on master.

Summary

• pt-heartbeat is rather useful in checking replica lag of MySQL and it also support PostgreSQL.
• pt-heartbeat will read connection options in my.cnf by default if you do not specify them.
• If you've executed the pt-heartbeat with option "--daemonize",don't forget to kill the redundant processes.
• We can retrieve these lag information in heartbeat table using our monitor tool or script.