Percona-Tookit工具包之pt-index-usage

 

Preface

 

    There're many ways relevent with performance tuning.For example,using indexes properly is important in doing that.At the very beginning of releasing a project,we're probably supposed to create many different indexes(especially union index) to increase the efficiency of queries on target tables even if some of them are seldom or never used at all.We are sure about that it is not the more the better of indexes on a table.Indexes will occupy more disk space and will cost a lot in maintaining.Alternatively,we should reduce the indexes which are not usually used by freqeuntly cheking them.Therefore,I'll introduce a tool which can help us in the aspect.

 

Introduce

 

    pt-index-usage(as what it is called) is a tool of Percona-Toolkit can provide a way to analyze your SQL statments in slow log(which means they're probably executed with bad performance).Afterward,you can know details about whether there're indexes not used properly and estimate whether to drop them in some time later.

 

Procedure

 

Usage

 pt-index-usage [OPTIONS] [FILES]

Main parameter

 --save-results-database -- Save output results into the specific tables of database.
 --create-save-results-database -- Create a database with necessary tables if set "--save-results-database" but not exist.
 --empty-save-results-tables -- Drop and recreate all the tables which are specified by "--save-results-database".
 --create-views -- Create views for tables in database which is specified by "--save-results-database".
 --no-report -- Don't generate a report but put results into tables for later analysis."--save-results-database" is indispensable when you set this option.
 --report-format -- The only format is "drop_unused_indexes" now.
 --drop -- Specify the type of index which you want to drop(Default value is non-unique).

Examples

 

Create test environment.

 (root@localhost mysql3306.sock)[zlm]::>create table if not exists test_index_usage(
     -> id int unsigned auto_increment not null,
     -> order_id int unsigned not null default ,
     -> name varchar() not null default '',
     -> gender enum('male','female') not null,
     -> primary key(id)
     -> ) auto_increment= engine=innodb charset=utf8mb4;
 Query OK,  rows affected (0.04 sec)

 (root@localhost mysql3306.sock)[zlm]::>delimiter $$
 (root@localhost mysql3306.sock)[zlm]::>create procedure pro_index_usage (in n1 int,in s1 varchar(),in s2 varchar())
     -> begin
     -> declare i int unsigned default ;
     -> start transaction;
     -> while i < n1 do
     -> insert into test_index_usage(order_id,gender,name) values(i,s1,s2);
     -> set i=i+;
     -> end while;
     -> commit;
     -> end;
     -> $$
 Query OK,  rows affected (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>delimiter ;
 (root@localhost mysql3306.sock)[zlm]::>call pro_index_usage(,'male','zlm');
 Query OK,  rows affected (5.59 sec)

 (root@localhost mysql3306.sock)[zlm]::>call pro_index_usage(,'female','aaron8219');
 Query OK,  rows affected (5.38 sec)

 (root@localhost mysql3306.sock)[zlm]::>select count(*) from test_index_usage;
 +----------+
 | count(*) |
 +----------+
 |    |
 +----------+
  row in set (0.05 sec)

 (root@localhost mysql3306.sock)[zlm]::>select * from test_index_usage limit ;
 +----+----------+------+--------+
 | id | order_id | name | gender |
 +----+----------+------+--------+
 |   |         | zlm  | male   |
 |   |         | zlm  | male   |
 |   |         | zlm  | male   |
 |   |         | zlm  | male   |
 |   |         | zlm  | male   |
 +----+----------+------+--------+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>alter table test_index_usage add key idx_key1 (order_id,gender);
 Query OK,  rows affected (0.64 sec)
 Records:   Duplicates:   Warnings: 

 (root@localhost mysql3306.sock)[zlm]::>alter table test_index_usage add key idx_key2 (order_id,gender,name);
 Query OK,  rows affected (0.94 sec)
 Records:   Duplicates:   Warnings: 

 (root@localhost mysql3306.sock)[zlm]::>show keys from test_index_usage;
 +------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
 | Table            | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
 +------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
 | test_index_usage |           | PRIMARY  |             | id          | A         |       |     NULL | NULL   |      | BTREE      |         |               |
 | test_index_usage |           | idx_key1 |             | order_id    | A         |       |     NULL | NULL   |      | BTREE      |         |               |
 | test_index_usage |           | idx_key1 |             | gender      | A         |       |     NULL | NULL   |      | BTREE      |         |               |
 | test_index_usage |           | idx_key2 |             | order_id    | A         |        |     NULL | NULL   |      | BTREE      |         |               |
 | test_index_usage |           | idx_key2 |             | gender      | A         |       |     NULL | NULL   |      | BTREE      |         |               |
 | test_index_usage |           | idx_key2 |             | name        | A         |       |     NULL | NULL   |      | BTREE      |         |               |
 +------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
  rows in set (0.00 sec)

Make sure "slow_query_on" has been set "on" and reduce the "long_query_time" into "0.01".

 (root@localhost mysql3306.sock)[zlm]::>show global variables like '%slow_query_log%';
 +---------------------+----------+
 | Variable_name       | Value    |
 +---------------------+----------+
 | slow_query_log      | ON       |
 | slow_query_log_file | slow.log |
 +---------------------+----------+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>show global variables like '%long_query_time%';
 +-----------------+----------+
 | Variable_name   | Value    |
 +-----------------+----------+
 | long_query_time | 1.000000 |
 +-----------------+----------+
  row in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>set global long_query_time=0.01;
 Query OK,  rows affected (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>show global variables like '%long_query_time%';
 +-----------------+----------+
 | Variable_name   | Value    |
 +-----------------+----------+
 | long_query_time | 0.010000 |
 +-----------------+----------+
  row in set (0.01 sec)

Execute a SQL statement.

 (root@localhost mysql3306.sock)[zlm]::>select * from test_index_usage where order_id>= and name='aaron8219';
 -- Omitted.
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 |  |     | aaron8219 | female |
 +--------+----------+-----------+--------+
  rows in set (0.16 sec)

Check the execute plan.

 (root@localhost mysql3306.sock)[zlm]::>explain select * from test_index_usage where order_id>= and name='aaron8219';
 +----+-------------+------------------+------------+-------+-------------------+----------+---------+------+-------+----------+--------------------------+
 | id | select_type | table            | partitions | type  | possible_keys     | key      | key_len | ref  | rows  | filtered | Extra                    |
 +----+-------------+------------------+------------+-------+-------------------+----------+---------+------+-------+----------+--------------------------+
 |   | SIMPLE      | test_index_usage | NULL       | range | idx_key1,idx_key2 | idx_key2 |        | NULL |  |    10.00 | Using where; Using index |
 +----+-------------+------------------+------------+-------+-------------------+----------+---------+------+-------+----------+--------------------------+
  row in set,  warning (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]::>explain format=json select * from test_index_usage where order_id>= and name='aaron8219'\G
 *************************** . row ***************************
 EXPLAIN: {
   "query_block": {
     "select_id": ,
     "cost_info": {
       "query_cost": "40540.88"
     },
     "table": {
       "table_name": "test_index_usage",
       "access_type": "range",
       "possible_keys": [
         "idx_key1",
         "idx_key2"
       ],
       "key": "idx_key2",
       "used_key_parts": [
         "order_id"
       ],
       "key_length": "",
       "rows_examined_per_scan": ,
       "rows_produced_per_join": ,
       "filtered": "10.00",
       "using_index": true,
       "cost_info": {
         "read_cost": "38544.88",
         "eval_cost": "1996.00",
         "prefix_cost": "40540.88",
         "data_read_per_join": "545K"
       },
       "used_columns": [
         "id",
         "order_id",
         "name",
         "gender"
       ],
       "attached_condition": "((`zlm`.`test_index_usage`.`order_id` >= 1) and (`zlm`.`test_index_usage`.`name` = 'aaron8219'))"
     }
   }
 }
  row in set,  warning (0.00 sec)

Check slow log.

 [root@zlm2 :: /data/mysql/mysql3306/data]
 #cat slow.log

 # Time: --25T08::.974728Z
 # User@Host: root[root] @ localhost []  Id:
 # Query_time: 0.161343  Lock_time: 0.000087 Rows_sent:   Rows_examined:
 SET timestamp=;
 select * from test_index_usage where order_id>= and name='aaron8219';

Execute pt-index-usage(create database and tables & views).

 [root@zlm2 :: /data/mysql/mysql3306/data]
 #pt-index-usage -h192.168.1. -P3306 -urepl -prepl4slave -Dzlm --create-views --no-report --create-save-results-database --save-results-database h=192.168.1.102,P=,u=repl,p=repl4slave,D=index_usage /data/mysql/mysql3306/data/slow.log

 [root@zlm2 :: /data/mysql/mysql3306/data]
 #

Check the tables & views in database "index_usage" on remote node zlm3.

 (root@localhost mysql3306.sock)[(none)]::>show databases;
 +--------------------+
 | Database           |
 +--------------------+
 | information_schema |
 | index_usage        |  -- This is the newly create database which contains tables created by pt-index-usage.
 | mysql              |
 | performance_schema |
 | sys                |
 | zlm                |
 +--------------------+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[(none)]::>use index_usage;
 Reading table information for completion of table and column names
 You can turn off this feature to get a quicker startup with -A

 Database changed
 (root@localhost mysql3306.sock)[index_usage]::>show tables;
 +---------------------------------+
 | Tables_in_index_usage           |
 +---------------------------------+
 | index_alternatives              |
 | index_usage                     |
 | indexes                         |
 | queries                         |
 | tables                          |
 | view_index_alternates           |
 | view_index_has_alternates       |
 | view_index_usage                |
 | view_query_uses_several_indexes |
 | view_required_indexes           |
 | view_unused_index_alternates    |
 +---------------------------------+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[index_usage]::>select * from index_alternatives;
 +---------------------+-----+------------------+----------+----------+-----+
 | query_id            | db  | tbl              | idx      | alt_idx  | cnt |
 +---------------------+-----+------------------+----------+----------+-----+
 |  | zlm | test_index_usage | idx_key2 | idx_key1 |    |
 +---------------------+-----+------------------+----------+----------+-----+
  row in set (0.00 sec)

 (root@localhost mysql3306.sock)[index_usage]::>select * from index_usage;
 +---------------------+-----+------------------+----------+-----+
 | query_id            | db  | tbl              | idx      | cnt |
 +---------------------+-----+------------------+----------+-----+
 |  | zlm | test_index_usage | idx_key2 |    |
 +---------------------+-----+------------------+----------+-----+
  row in set (0.00 sec)

 (root@localhost mysql3306.sock)[index_usage]::>select * from indexes;
 +-------+---------------------------+-----------+-----+
 | db    | tbl                       | idx       | cnt |
 +-------+---------------------------+-----------+-----+
 | mysql | columns_priv              | PRIMARY   |    |
 | mysql | db                        | PRIMARY   |    |
 | mysql | db                        | User      |    |
 | mysql | engine_cost               | PRIMARY   |    |
 | mysql | event                     | PRIMARY   |    |
 | mysql | func                      | PRIMARY   |    |
 | mysql | help_category             | name      |    |
 | mysql | help_category             | PRIMARY   |    |
 | mysql | help_keyword              | name      |    |
 | mysql | help_keyword              | PRIMARY   |    |
 | mysql | help_relation             | PRIMARY   |    |
 | mysql | help_topic                | name      |    |
 | mysql | help_topic                | PRIMARY   |    |
 | mysql | ndb_binlog_index          | PRIMARY   |    |
 | mysql | plugin                    | PRIMARY   |    |
 | mysql | proc                      | PRIMARY   |    |
 | mysql | procs_priv                | Grantor   |    |
 | mysql | procs_priv                | PRIMARY   |    |
 | mysql | proxies_priv              | Grantor   |    |
 | mysql | proxies_priv              | PRIMARY   |    |
 | mysql | servers                   | PRIMARY   |    |
 | mysql | server_cost               | PRIMARY   |    |
 | mysql | tables_priv               | Grantor   |    |
 | mysql | tables_priv               | PRIMARY   |    |
 | mysql | time_zone                 | PRIMARY   |    |
 | mysql | time_zone_leap_second     | PRIMARY   |    |
 | mysql | time_zone_name            | PRIMARY   |    |
 | mysql | time_zone_transition      | PRIMARY   |    |
 | mysql | time_zone_transition_type | PRIMARY   |    |
 | mysql | user                      | PRIMARY   |    |
 | sys   | sys_config                | PRIMARY   |    |
 | zlm   | checksums                 | PRIMARY   |    |
 | zlm   | checksums                 | ts_db_tbl |    |
 | zlm   | indexes                   | PRIMARY   |    |
 | zlm   | index_alternatives        | db        |    |
 | zlm   | index_alternatives        | db_2      |    |
 | zlm   | index_alternatives        | query_id  |    |
 | zlm   | index_usage               | query_id  |    |
 | zlm   | queries                   | PRIMARY   |    |
 | zlm   | tables                    | PRIMARY   |    |
 | zlm   | test_ddl                  | PRIMARY   |    |
 | zlm   | test_index_usage          | idx_key1  |    |
 | zlm   | test_index_usage          | idx_key2  |    |
 | zlm   | test_index_usage          | PRIMARY   |    |
 | zlm   | test_innodb               | PRIMARY   |    |
 | zlm   | test_myisam               | PRIMARY   |    |
 +-------+---------------------------+-----------+-----+
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[index_usage]::>select * from queries\G
 *************************** . row ***************************
    query_id:
 fingerprint: create table if not exists tables ( db varchar(?) not ?, tbl varchar(?) not ?, cnt bigint unsigned not ? default ?, primary key (db, tbl) )
      sample: CREATE TABLE IF NOT EXISTS tables (
     db           VARCHAR() NOT NULL,
     tbl          VARCHAR() NOT NULL,
     cnt          BIGINT UNSIGNED NOT NULL DEFAULT ,
     PRIMARY KEY  (db, tbl)
   )
 *************************** . row ***************************
    query_id:
 fingerprint: select * from test_index_usage where order_id>=? and name=?
      sample: select * from test_index_usage where order_id>= and name='aaron8219'
 *************************** . row ***************************
    query_id:
 fingerprint: create table if not exists indexes ( db varchar(?) not ?, tbl varchar(?) not ?, idx varchar(?) not ?, cnt bigint unsigned not ? default ?, primary key (db, tbl, idx) )
      sample: CREATE TABLE IF NOT EXISTS indexes (
     db           VARCHAR() NOT NULL,
     tbl          VARCHAR() NOT NULL,
     idx          VARCHAR() NOT NULL,
     cnt          BIGINT UNSIGNED NOT NULL DEFAULT ,
     PRIMARY KEY  (db, tbl, idx)
   )
 *************************** . row ***************************
    query_id:
 fingerprint: create table if not exists queries ( query_id bigint unsigned not ?, fingerprint text not ?, sample text not ?, primary key (query_id) )
      sample: CREATE TABLE IF NOT EXISTS queries (
     query_id     BIGINT UNSIGNED NOT NULL,
     fingerprint  TEXT NOT NULL,
     sample       TEXT NOT NULL,
     PRIMARY KEY  (query_id)
   )
 *************************** . row ***************************
    query_id:
 fingerprint: create table if not exists index_alternatives ( query_id bigint unsigned not ?, db varchar(?) not ?, tbl varchar(?) not ?, idx varchar(?) not ?, alt_idx varchar(?) not ?, cnt bigint unsigned not ? default ?, unique index (query_id, db, tbl, idx, alt_idx), index (db, tbl, idx), index (db, tbl, alt_idx) )
      sample: CREATE TABLE IF NOT EXISTS index_alternatives (
     query_id      BIGINT UNSIGNED NOT NULL, -- This query used
     db            VARCHAR() NOT NULL,     -- this index, but...
     tbl           VARCHAR() NOT NULL,     --
     idx           VARCHAR() NOT NULL,     --
     alt_idx       VARCHAR() NOT NULL,     -- was an alternative
     cnt           BIGINT UNSIGNED NOT NULL DEFAULT ,
     UNIQUE INDEX  (query_id, db, tbl, idx, alt_idx),
     INDEX         (db, tbl, idx),
     INDEX         (db, tbl, alt_idx)
   )
  rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[index_usage]::>select * from tables;
 +-------+---------------------------+-----+
 | db    | tbl                       | cnt |
 +-------+---------------------------+-----+
 | mysql | columns_priv              |    |
 | mysql | db                        |    |
 | mysql | engine_cost               |    |
 | mysql | event                     |    |
 | mysql | func                      |    |
 | mysql | help_category             |    |
 | mysql | help_keyword              |    |
 | mysql | help_relation             |    |
 | mysql | help_topic                |    |
 | mysql | ndb_binlog_index          |    |
 | mysql | plugin                    |    |
 | mysql | proc                      |    |
 | mysql | procs_priv                |    |
 | mysql | proxies_priv              |    |
 | mysql | servers                   |    |
 | mysql | server_cost               |    |
 | mysql | tables_priv               |    |
 | mysql | time_zone                 |    |
 | mysql | time_zone_leap_second     |    |
 | mysql | time_zone_name            |    |
 | mysql | time_zone_transition      |    |
 | mysql | time_zone_transition_type |    |
 | mysql | user                      |    |
 | sys   | sys_config                |    |
 | zlm   | checksums                 |    |
 | zlm   | indexes                   |    |
 | zlm   | index_alternatives        |    |
 | zlm   | index_usage               |    |
 | zlm   | queries                   |    |
 | zlm   | tables                    |    |
 | zlm   | test_ddl                  |    |
 | zlm   | test_ddl_no_pk            |    |
 | zlm   | test_index_usage          |    |
 | zlm   | test_innodb               |    |
 | zlm   | test_myisam               |    |
 +-------+---------------------------+-----+
  rows in set (0.00 sec)

Execute pt-index-usage(output on screen derectly).

 [root@zlm2 :: /data/mysql/mysql3306/data]
 #pt-index-usage -h192.168.1. -P3306 -urepl -prepl4slave -Dzlm /data/mysql/mysql3306/data/slow.log

 ALTER TABLE `zlm`.`test_index_usage` DROP KEY `idx_key1`; -- type:non-unique

 [root@zlm2 :: /data/mysql/mysql3306/data]
 #

summary

• pt-index-usage provides merely analysis on slow log at the moment.
• Usually We'll analyze SQL statements in slow log,and then put them into tables of database on remote server(which maybe not slave).
• There will be a series of insert operations when using pt-index-usage(while not setting "--no-report").Thus,it really will I ncreast the load of product server.
  
• On the other hand,It's possible to be misled when we get a probable bad execute plan which bypasses the useful indexes.