执行SQL查询导致磁盘耗尽故障演示

a fellow in IMG wechat group 2 met an error about running out of disk space when using MySQL query today,now let's mimic the situation:

first,i'll create a test table and a procedure,then using the procedure to insert 1000W records into test table "test":

(root@localhost mysql3306.sock)[zlm]08:46:36>create table test(

    -> id int unsigned not null,  --Notice,there is no primary key and any other key here.

    -> name char(50) not null default ''

    -> ) engine=innodb charset=utf8;

Query OK, 0 rows affected (0.01 sec)

(root@localhost mysql3306.sock)[zlm]08:46:45>delimiter $$

(root@localhost mysql3306.sock)[zlm]08:46:54>create procedure pro_test()

    -> begin

    -> declare i int;

    -> set i=10000000;

    -> while i>0 do

    -> insert into test(id,name) values (i,'aaron8219');

    -> set i=i-1;

    -> end while;

    -> end $$

Query OK, 0 rows affected (0.00 sec)

(root@localhost mysql3306.sock)[zlm]08:46:54>delimiter ;

(root@localhost mysql3306.sock)[zlm]08:47:13>call pro_test;

^C^C -- query aborted  --I'm afraid the disk space will be insufficient,so i cancel the procedure by Ctrl+C.

Terminated

it's the incremental change result of the command "df -h" output below at the begining to the point that i cancel insertation:

 [root@zlm3 08:47:20 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.1G  3.4G  60% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:47:22 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.8G  2.6G  70% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:51:27 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  6.2G  2.3G  74% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:53:06 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  7.0G  1.4G  84% /  -- The free disk space became 16% (maybe less) when i cancel the operation.

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   77G  9.6G  89% /vagrant

 [root@zlm3 08:57:30 /data/mysql/mysql3306]

 #ps -ef|grep mysql

 mysql     6031     1 12 08:00 ?        00:07:05 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf

 root      6182  6157  0 08:46 pts/0    00:00:00 mysql

 root      6389  6085  0 08:58 pts/1    00:00:00 grep --color=auto mysql

 [root@zlm3 08:58:13 /data/mysql/mysql3306]

 #kill 6182  -- After cancel the operation,i kill the mysql process in this session.

 [root@zlm3 08:58:22 /data/mysql/mysql3306]

 #ps -ef|grep mysql

 mysql     6031     1 12 08:00 ?        00:07:17 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf

 root      6403  6085  0 08:58 pts/1    00:00:00 grep --color=auto mysql

 [root@zlm3 08:58:29 /data/mysql/mysql3306]

 #

in the first session,relogin with mysql client,check the status of the table test:

 #mysql

 Welcome to the MySQL monitor.  Commands end with ; or \g.

 Your MySQL connection id is 7

 Server version: 5.7.21-log MySQL Community Server (GPL)

 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

 Oracle is a registered trademark of Oracle Corporation and/or its

 affiliates. Other names may be trademarks of their respective

 owners.

 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 (root@localhost mysql3306.sock)[(none)]08:58:35>use zlm;

 Reading table information for completion of table and column names

 You can turn off this feature to get a quicker startup with -A

 Database changed

 (root@localhost mysql3306.sock)[zlm]08:59:08>show tables;

 +---------------+

 | Tables_in_zlm |

 +---------------+

 | t1            |

 | t2            |

 | t3            |

 | test          |

 +---------------+

 4 rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]09:00:49>select table_schema,concat(truncate(sum(data_length)/1024/1024,2),'MB') as data_size,

     -> concat(truncate(sum(index_length)/1024/1024,2),'MB') as index_size

     -> from information_schema.tables

     -> where table_schema='zlm' and table_name='test';

 +--------------+-----------+------------+

 | table_schema | data_size | index_size |

 +--------------+-----------+------------+

 | zlm          | 656.00MB  | 0.00MB     |  --Now the table "test" has alread been a big table,the size turned into 656Mb.

 +--------------+-----------+------------+

 1 row in set (0.10 sec)

 (root@localhost mysql3306.sock)[zlm]09:01:34>select count(*) from test;

 +----------+

 | count(*) |

 +----------+

 |  9070823 | --Almost 1000W records around.

 +----------+

 1 row in set (19.16 sec)

 (root@localhost mysql3306.sock)[zlm]09:02:29>show master status;

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 | File             | Position  | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                               |

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 | mysql-bin.000057 | 244082592 |              |                  | 5c77c31b-4add-11e8-81e2-080027de0e0e:1-13527727 |

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 1 row in set (0.06 sec)

 (root@localhost mysql3306.sock)[zlm]09:12:58>

let's check the binlog file,these files really ocuppied a huge amount disk space:

[root@zlm3 08:58:42 /data/mysql/mysql3306/logs]

#ls -l

total 2384300

-rw-r----- 1 mysql mysql      3831 May 28 03:20 mysql-bin.000019

-rw-r----- 1 mysql mysql      9564 May 28 11:37 mysql-bin.000020

-rw-r----- 1 mysql mysql      4761 May 29 11:27 mysql-bin.000021

-rw-r----- 1 mysql mysql       217 May 30 11:29 mysql-bin.000022

-rw-r----- 1 mysql mysql       217 May 31 03:20 mysql-bin.000023

-rw-r----- 1 mysql mysql       613 May 31 03:29 mysql-bin.000024

-rw-r----- 1 mysql mysql      1009 May 31 03:35 mysql-bin.000025

-rw-r----- 1 mysql mysql       217 May 31 03:36 mysql-bin.000026

-rw-r----- 1 mysql mysql       217 May 31 03:37 mysql-bin.000027

-rw-r----- 1 mysql mysql       217 May 31 03:38 mysql-bin.000028

-rw-r----- 1 mysql mysql       217 May 31 03:40 mysql-bin.000029

-rw-r----- 1 mysql mysql      1563 May 31 03:45 mysql-bin.000030

-rw-r----- 1 mysql mysql       217 May 31 06:50 mysql-bin.000031

-rw-r----- 1 mysql mysql       217 May 31 06:59 mysql-bin.000032

-rw-r----- 1 mysql mysql       217 May 31 07:04 mysql-bin.000033

-rw-r----- 1 mysql mysql       217 May 31 07:13 mysql-bin.000034

-rw-r----- 1 mysql mysql       217 May 31 07:15 mysql-bin.000035

-rw-r----- 1 mysql mysql       217 May 31 07:42 mysql-bin.000036

-rw-r----- 1 mysql mysql       461 May 31 08:22 mysql-bin.000037

-rw-r----- 1 mysql mysql       217 May 31 08:25 mysql-bin.000038

-rw-r----- 1 mysql mysql       613 May 31 10:37 mysql-bin.000039

-rw-r----- 1 mysql mysql       369 May 31 10:41 mysql-bin.000040

-rw-r----- 1 mysql mysql       613 May 31 11:28 mysql-bin.000041

-rw-r----- 1 mysql mysql      3141 Jun  1 10:10 mysql-bin.000042

-rw-r----- 1 mysql mysql      5677 Jun  1 11:38 mysql-bin.000043

-rw-r----- 1 mysql mysql       217 Jun  4 07:54 mysql-bin.000044

-rw-r----- 1 mysql mysql       194 Jun  4 07:57 mysql-bin.000045

-rw-r----- 1 mysql mysql       217 Jun  4 07:57 mysql-bin.000046

-rw-r----- 1 mysql mysql       217 Jun  4 11:23 mysql-bin.000047

-rw-r----- 1 mysql mysql 268435609 Jun  5 08:48 mysql-bin.000048

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:50 mysql-bin.000049

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:52 mysql-bin.000050

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:54 mysql-bin.000051

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:55 mysql-bin.000052

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:57 mysql-bin.000053

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:58 mysql-bin.000054

-rw-r----- 1 mysql mysql 268435737 Jun  5 09:00 mysql-bin.000055

-rw-r----- 1 mysql mysql 268435737 Jun  5 09:02 mysql-bin.000056  --From binlog 48 to 56,each one exhausted the max size at 256M.

-rw-r----- 1 mysql mysql  25366220 Jun  5 09:02 mysql-bin.000057  --This is the last binlog file contain the last several records.

-rw-r----- 1 mysql mysql      1716 Jun  5 09:02 mysql-bin.index

[root@zlm3 09:09:43 /data/mysql/mysql3306/logs]

#mysqlbinlog -vv --base64-output=decode-rows mysql-bin.000057| tail -20

#180605  9:03:45 server id 1023306  end_log_pos 244082462 CRC32 0x9db85944     Query    thread_id=5    exec_time=0    error_code=0

SET TIMESTAMP=1528182225/*!*/;

BEGIN

/*!*/;

# at 244082462

#180605  9:03:45 server id 1023306  end_log_pos 244082511 CRC32 0x8bb85bae     Table_map: `zlm`.`test` mapped to number 104

# at 244082511

#180605  9:03:45 server id 1023306  end_log_pos 244082561 CRC32 0x6ede8301     Write_rows: table id 104 flags: STMT_END_F

### INSERT INTO `zlm`.`test`

### SET

###   @1=1 /* INT meta=0 nullable=0 is_null=0 */

###   @2='aaron8219' /* STRING(150) meta=65174 nullable=0 is_null=0 */

# at 244082561

#180605  9:03:45 server id 1023306  end_log_pos 244082592 CRC32 0xf983b21b     Xid = 30000069  --The "end_log_pos 244082592" is equal with the output result of the postion in "show master status;" command.

COMMIT/*!*/;

SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; --GTID_NEXT='AUTOMATIC' means there're no more sequential binlogs behind.

DELIMITER ;

# End of log file

/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;

/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

[root@zlm3 09:11:21 /data/mysql/mysql3306/logs]

#

now,let's do some testing query below(using distinct/group by):

 (root@localhost mysql3306.sock)[zlm]09:16:31>select count(distinct(id)) from test group by name;

 ERROR 3 (HY000): Error writing file '/data/mysql/mysql3306/tmp/MYCUMGkU' (Errcode: 28 - No space left on device)

 (root@localhost mysql3306.sock)[zlm]09:18:34>system ps aux|grep mysql

 root      6420  0.0  0.2 134112  2292 pts/0    S+   08:58   0:00 mysql

 root      6667  0.0  0.1 113116  1360 pts/0    S+   09:17   0:00 sh -c  ps aux|grep mysql

 root      6669  0.0  0.0 112640   944 pts/0    R+   09:17   0:00 grep mysql

 (root@localhost mysql3306.sock)[zlm]10:22:13>

eventually,the query ended with error "Errcode:28 - no space left on device",obviously the mysqld process has dead now.check the disk space,it's 99% in Use% column,why does the disk space continuously increase?let's see the official document about this bellow:

8.4.4 Internal Temporary Table Use in MySQL

In some cases, the server creates internal temporary tables while processing statements. Users have no direct control over when this occurs.

The server creates temporary tables under conditions such as these:

Evaluation of UNION statements, with some exceptions described later.
Evaluation of some views, such those that use the TEMPTABLE algorithm, UNION, or aggregation.
Evaluation of derived tables (see Section 13.2.11.8, “Derived Tables”).
Evaluation of common table expressions (see Section 13.2.13, “WITH Syntax (Common Table Expressions)”).
Tables created for subquery or semi-join materialization (see Section 8.2.2, “Optimizing Subqueries, Derived Tables, View References, and Common Table Expressions”).
Evaluation of statements that contain an ORDER BY clause and a different GROUP BY clause, or for which the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue.
Evaluation of DISTINCT combined with ORDER BY may require a temporary table.
For queries that use the SQL_SMALL_RESULT modifier, MySQL uses an in-memory temporary table, unless the query also contains elements (described later) that require on-disk storage.
To evaluate INSERT ... SELECT statements that select from and insert into the same table, MySQL creates an internal temporary table to hold the rows from the SELECT, then inserts those rows into the target table. See Section 13.2.6.1, “INSERT ... SELECT Syntax”.
Evaluation of multiple-table UPDATE statements.
Evaluation of GROUP_CONCAT() or COUNT(DISTINCT) expressions.

来源： https://dev.mysql.com/doc/refman/8.0/en/internal-temporary-tables.html

here's the conclusion,whenever above conditions occured,it will induce MySQL server to do sort operation in memory or in disk(this is too bad thing),hence,in our test above,the disk space was exhausted soon.

 [root@zlm3 08:58:29 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  122M  99% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 09:19:20 /data/mysql/mysql3306]

 #du -sh /data

 3.8G    /data --Even though the "df -h" shows that it's out of space already,but the "/data/" dir still don't increase at all.

 [root@zlm3 09:20:21 /data/mysql/mysql3306]

 #mysqladmin var|grep -i tmp --Check the parameter about key word "tmp".

 | default_tmp_storage_engine                               | InnoDB                                                                                                                                                                                                                                                                                                                                                                                                           |

 | innodb_temp_data_file_path                               | ibtmp1:12M:autoextend                                                                                                                                                                                                                                                                                                                                                                                            |

 | innodb_tmpdir                                            |                                                                                                                                                                                                                                                                                                                                                                                                                  |

 | internal_tmp_disk_storage_engine                         | InnoDB                                                                                                                                                                                                                                                                                                                                                                                                           |

 | max_tmp_tables                                           | 32                                                                                                                                                                                                                                                                                                                                                                                                               |

 | slave_load_tmpdir                                        | /data/mysql/mysql3306/tmp                                                                                                                                                                                                                                                                                                                                                                                        |

 | socket                                                   | /tmp/mysql3306.sock                                                                                                                                                                                                                                                                                                                                                                                              |

 | tmp_table_size                                           | 100663296                                                                                                                                                                                                                                                                                                                                                                                                        |

 | tmpdir                                                   | /data/mysql/mysql3306/tmp                                                                                                                                                                                                                                                                                                                                                                                        |

 [root@zlm3 09:22:54 /data/mysql/mysql3306/tmp]

 #ls -l --No files here at all,why?'cause it has been deleted,see below:

 total 0

 [root@zlm3 09:22:56 /data/mysql/mysql3306/tmp]

 #lsof -n|grep deleted --List open files using "lsof" command.there're lots of deleted tmp file used by mysqld process

 tuned      566         root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 gmain      566  719    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  721    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  723    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  728    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 mysqld    6031        mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031        mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031        mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031        mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6036   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6036   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6036   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6036   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6037   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6037   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6037   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6037   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6038   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6038   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6038   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6038   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6039   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6039   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6039   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6039   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6040   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6040   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6040   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6040   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6041   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6041   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6041   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6041   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6042   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6042   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6042   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6042   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6043   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6043   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6043   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6043   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6044   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6044   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6044   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6044   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6045   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6045   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6045   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6045   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6046   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6046   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6046   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6046   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6047   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6047   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6047   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6047   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6049   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6049   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6049   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6049   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6050   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6050   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6050   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6050   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6051   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6051   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6051   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6051   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6052   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6052   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6052   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6052   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6053   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6053   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6053   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6053   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6054   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6054   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6054   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6054   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6055   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6055   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6055   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6055   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6056   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6056   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6056   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6056   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6057   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6057   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6057   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6057   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6058   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6058   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6058   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6058   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6059   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6059   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6059   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6059   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6060   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6060   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6060   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6060   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6061   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6061   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6061   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6061   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6062   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6062   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6062   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6062   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6064   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6064   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6064   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6064   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6387   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6387   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6387   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6387   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 [root@zlm3 09:24:32 /data/mysql/mysql3306/tmp]

 #lsof -n | grep deleted | awk -F ' ' '{printf "kill -9 %s\n" $2}' > /root/free_handle_space.sh

 awk: cmd. line:1: (FILENAME=- FNR=1) fatal: not enough arguments to satisfy format string

     `kill -9 %s

 566'

              ^ ran out for this one --Is this a gramma mistake here?i've been tested before and it's ok,should it be the lack of disk space?

 [root@zlm3 09:53:07 /data/mysql/mysql3306/tmp]

 #

the process id here is 6031,i supposed that i can kill the process to release the handler of files and get back the disk space:

 [root@zlm3 09:56:06 /data/mysql/mysql3306/tmp]

 #kill 6031

 [root@zlm3 10:13:42 /data/mysql/mysql3306/tmp]

 #lsof -n | grep deleted  --The lots of deleted tmp file has gone now.

 tuned      566         root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 gmain      566  719    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  721    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  723    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  728    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 [root@zlm3 10:14:06 /data/mysql/mysql3306/tmp]

 #

what shocked me is that the disk space was not released after my killing operation:

 [root@zlm3 10:14:17 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  134M  99% /  --Still 99% in Use% cloumn

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:15:25 ~]

 #

what's wrong with it?what if i kill the mysql process?

 [root@zlm3 10:14:06 /data/mysql/mysql3306/tmp]

 #ps aux|grep mysql

 root      6420  0.0  0.2 134112  2292 pts/0    S+   08:58   0:00 mysql

 root      6656  0.0  0.0 112640   960 pts/1    R+   10:21   0:00 grep --color=auto mysql

 [root@zlm3 10:21:31 /data/mysql/mysql3306/tmp]

 #kill 6420

 [root@zlm3 10:29:39 /data/mysql/mysql3306/tmp]

 #ps aux|grep mysql

 root      6679  0.0  0.0 112640   960 pts/1    R+   10:29   0:00 grep --color=auto mysql

 [root@zlm3 10:29:49 /data/mysql/mysql3306/tmp]

 #

nothing changed,the disk space is still full:

 [root@zlm3 10:29:47 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  134M  99% /  --Change nothing,99% again.

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:29:55 ~]

 #

after i drop the table "test",it just release 10% disk sapce,where has the free space gone?

 [root@zlm3 10:29:55 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  7.5G  962M  89% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:34:16 ~]

 #

then,i use "reset master;",the dike space finally came back:

 (root@localhost mysql3306.sock)[zlm]10:34:10>reset master;  --It's utterly forbidden to use this command on product MySQL server,take care.

 Query OK, 0 rows affected (0.08 sec)

 (root@localhost mysql3306.sock)[zlm]10:41:44>show tables;

 +---------------+

 | Tables_in_zlm |

 +---------------+

 | t1            |

 | t2            |

 | t3            |

 +---------------+

 3 rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]10:42:00>

 [root@zlm3 10:34:16 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.0G  3.5G  60% /  --Now,all the space came back(before the insertation).

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:41:50 ~]

 #cd /data/mysql/mysql3306/logs

 [root@zlm3 10:47:49 /data/mysql/mysql3306/logs]

 #ls -l

 total 8

 -rw-r----- 1 mysql mysql 154 Jun  5 10:41 mysql-bin.000001

 -rw-r----- 1 mysql mysql  44 Jun  5 10:41 mysql-bin.index

 [root@zlm3 10:47:50 /data/mysql/mysql3306/logs]

 #

Summary:

the test what i made just now was not so good to explain what i wanna say,perhaps i didn't use the accurate sample data.
generally speaking,the file handler is the common reason why the deletion operation doesn't release disk space.
operations such as "distinct","group by" in MySQL query will use temperory table to sort on disk,it depend on the several parameter("max_tmp_tables","max_heap_table_size" and etc.)you've set in config file "my.cnf".
not using index will lead to huge amount of sort operation which will generate poor performance of MySQL server.
before you kill the process to release the disk space,do think twice that the deleted file is really unnecessary any more.

执行SQL查询导致磁盘耗尽故障演示的更多相关文章

EF5中执行 sql语句使用Database.ExecuteSqlCommand 返回影响的行数 ; EF5执行sql查询语句 Database.SqlQuery 带返回值
一: 执行sql语句,返回受影响的行数在mysql里面,如果没有影响,那么返回行数为 -1 ,sqlserver 里面还没有测试过 using (var ctx = new MyDbConte ...
Oracle执行SQL查询语句的步骤
Oracle执行SQL查询语句的步骤如果用户在SQL*Plus下输入了如下查询语句:SELECT * FROM dept: 查询语句的处理主要包括三个过程:编译(parse).执行(execute) ...
oracle执行sql查询语句出现错误ORA-00942：表或视图不存在
情况是这样,A库的用户名和表空间分别为SH , SH 把业务表SH所有数据从A库,导入到B库, 表空间为SH,用户名为SP 在B库里面执行sql查询语句出现错误ORA-00942:表或视图不存在语句 ...
ado执行sql查询出现“发送数据流时出现算术溢出”错误
开发一个数据采集监控系统,比较变态的是有将近2000项数据.根据数据类型分多个表存储.数据库访问层采用ado.最近发现当一条sql一次性查询1700多个字段数据后就出现“发送数据流时出现算术溢出”错误 ...
MySQL数据库详解（一）执行SQL查询语句时,其底层到底经历了什么?
一条SQL查询语句是如何执行的? 前言大家好,我是WZY,今天我们学习下MySQL的基础框架,看一件事千万不要直接陷入细节里,你应该先鸟瞰其全貌,这样能够帮助你从高维度理解问题.同样,对于MyS ...
python MySQL执行SQL查询结果返回字典
写自动化测试的时候我希望执行数据库前置任务,把数据库查询的结果作为请求的参数,但是正常返回结果为列表嵌套里面,这样就会影响到关键字准确的获取,特别的受限于SQL的查询字段的的顺序,所以希望返回的单条数 ...
MySQL执行sql查询并上传至远程服务器
最近项目中有需要做一个shell脚本,可以对一个数据库执行sql操作,并将结果转为txt,筛选结果用tab隔开,保存至一个远程服务器上,以供其他人用Excel读取用txt中的内容. MySQL中将结果 ...
Oracle 在函数或存储过程中执行sql查询字符串并将结果值赋值给变量
请看黄色部分 --区县指标 THEN TVALUE_SQL := 'SELECT TO_CHAR(' || CUR_ROW.MAIN_FIELD || ') FROM ' || CUR_ROW.END ...
HBase 查询导致RegionServer OOM故障复盘
背景:我司作为某运营商公司的技术咨询公司,发现有第三方开发公司在使用HBase 1.1.2 (HDP 2.4.2.258版本)一段时间使用正常后,从某一天开始报OOM,从而导致RegionServer ...

随机推荐

File中mkdir()和mkdirs()的区别
mkdir() 创建此抽象路径名指定的目录.只能在已经存在的目录中创建文件夹如: File folder = new File("d:\\test1\\test2"); fold ...
转：hive面试题
有一张很大的表:TRLOG该表大概有2T左右TRLOG:CREATE TABLE TRLOG(PLATFORM string,USER_ID int,CLICK_TIME string,CLICK_U ...
【Python】面向对象编程思想
概念 "笔"作为一个抽象的概念,可以被看成是一个类.而一支实实在在的笔,则是"笔"这种类型的对象. 一个类可以有属于它的函数,这种函数被称为类的"方法 ...
atom 常用配置
基本配置 setting 位于 File -> setting 显示HTML标签闭合的竖线 Setting -> Editor Setting -> 勾选 Show Indent G ...
关于ios 10 的新的部分
编译器Xcode 8.2.1 SDK 10.2 1. 关于麦克风,相机,相册等部分的权限,需要在info.plist内进行设置,否则会出现crash.
如何判断单链表是否存在环 & 判断两链表是否相交
给定一个单链表,只给出头指针h: 1.如何判断是否存在环? 2.如何知道环的长度? 3.如何找出环的连接点在哪里? 4.带环链表的长度是多少? 解法: 1.对于问题1,使用追赶的方法,设定两个指针sl ...
MySQL存储过程和临时表
MySQL创建存储过程 MySQL中,创建存储过程的基本形式如下: CREATE PROCEDURE sp_name ([proc_parameter[,...]]) [characteristic ...
新款 2018款macbook Pro 装双系统教程
首个阅读量将破万的文章,感谢支持.防止无良爬虫,开头附上原文链接:http://www.cnblogs.com/xueyudlut/p/7498115.html ------------------- ...
July 21st 2017 Week 29th Friday
If you want to fly too high in relation to the horizon forget. 要想飞得高,就该把地平线忘掉. Always keep our eyes ...
java中i = i++问题
今天偶然呗问到了一个java中自增运算符的问题,涉及到了堆栈,觉得挺有趣的把它记录下来. int i=0; i=i++; 结果i是多少? 这是一个经常被提及的问题,答案一 ...

执行SQL查询导致磁盘耗尽故障演示

8.4.4 Internal Temporary Table Use in MySQL

执行SQL查询导致磁盘耗尽故障演示的更多相关文章

随机推荐

热门专题