记一次mysql故障恢复

事情要从俩月前的一个坑说起，一台新的测试服务器，新项目一元夺宝用的。

配置aws上的一台云主机，系统盘8G，一块300G的云硬盘。

拿到机器后，另一运维小哥安装php，nginx，mysql等软件。 安装完后，然忘了挂载云硬盘。 随着根分区日志越来越多，mysql数据量增大。。。磁盘已满，mysql自行挂了。。。哭。。

开发找上门来，mysql插不进去了，我连上服务器，ps -ef。。 没有进程。

尝试重启。。失败。

查看错误日志。。如下：

 161206 08:07:41 mysqld_safe Starting mysqld daemon with databases from /data/mysql
 2016-12-06 08:07:41 4473 [Note] InnoDB: The InnoDB memory heap is disabled
 2016-12-06 08:07:41 4473 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 2016-12-06 08:07:41 4473 [Note] InnoDB: Compressed tables use zlib 1.2.3
 2016-12-06 08:07:41 4473 [Note] InnoDB: Using Linux native AIO
 2016-12-06 08:07:41 4473 [Note] InnoDB: Using CPU crc32 instructions
 2016-12-06 08:07:41 4473 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 2016-12-06 08:07:41 4473 [Note] InnoDB: Completed initialization of buffer pool
 2016-12-06 08:07:41 4473 [Note] InnoDB: Highest supported file format is Barracuda.
 2016-12-06 08:07:41 4473 [Note] InnoDB: Log scan progressed past the checkpoint lsn 416493596
 2016-12-06 08:07:41 4473 [Note] InnoDB: Database was not shutdown normally!
 2016-12-06 08:07:41 4473 [Note] InnoDB: Starting crash recovery.
 2016-12-06 08:07:41 4473 [Note] InnoDB: Reading tablespace information from the .ibd files...
 2016-12-06 08:07:41 4473 [Note] InnoDB: Restoring possible half-written data pages
 2016-12-06 08:07:41 4473 [Note] InnoDB: from the doublewrite buffer...
 InnoDB: Doing recovery: scanned up to log sequence number 416570218
 2016-12-06 08:07:41 4473 [Note] InnoDB: Starting an apply batch of log records to the database...
 InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
 InnoDB: Apply batch completed
 InnoDB: Last MySQL binlog file position 0 177880661, file name mysql-bin.000020
 2016-12-06 08:07:42 4473 [Note] InnoDB: 128 rollback segment(s) are active.
 2016-12-06 08:07:42 4473 [Note] InnoDB: Waiting for purge to start
 2016-12-06 08:07:42 4473 [Note] InnoDB: 5.6.14 started; log sequence number 416570218
 /usr/local/app/mysql-5.6.14/bin/mysqld: Error writing file '/data/mysql/mysql-slow.log' (Errcode: 28 - No space left on device)
 2016-12-06 08:07:42 4473 [ERROR] Could not use /data/mysql/mysql-slow.log for logging (error 28). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
 08:07:42 UTC - mysqld got signal 11 ;
 This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
 We will try our best to scrape up some info that will hopefully help
 diagnose the problem, but since we have already crashed,
 something is definitely wrong and this may fail.

 key_buffer_size=33554432
 read_buffer_size=33554432
 max_used_connections=0
 max_threads=1000
 thread_count=0
 connection_count=0
 It is possible that mysqld could use up to
 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 65582393 K  bytes of memory
 Hope that's ok; if not, decrease some variables in the equation.

 Thread pointer: 0x0
 Attempting backtrace. You can use the following information to find out
 where mysqld died. If you see no messages after this, something went
 terribly wrong...
 stack_bottom = 0 thread_stack 0x40000
 /usr/local/app/mysql-5.6.14/bin/mysqld(my_print_stacktrace+0x35)[0x8dd875]
 /usr/local/app/mysql-5.6.14/bin/mysqld(handle_fatal_signal+0x40b)[0x64ddbb]
 /lib64/libpthread.so.0(+0xf7e0)[0x7f926e2b77e0]
 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 information that should help you find out what is causing the crash.
 161206 08:07:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

第25行亮了。No space left on device。找到问题了，df一看，根分区使用率100%。。。剩下/app目录300G使用率1%在那孤零零挂着。

解决办法：

移动原有文件到数据分区

# mv /data/mysql  /app/

将

datadir = /data/mysql 改为datadir = /app/mysql

重启正常。