cvm母机宕机重启后数据库修复

下午正在开周会，然后收到短信，说是X.X.X.X的机器ping不通了，一轮测试过后，发现是某台数据库服务器挂了，先不急着重启，问下tencent客服。。。

乖乖的好家伙，母机的主板坏了。。。。一个小时候，母机起来了，看下数据库起来么。。

[root@VM_145_57_tlinux ~]# mysql -uroot -p1234
ERROR  (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
[root@VM_145_57_tlinux ~]# mysql -uroot -p
Enter password:
ERROR  (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
[root@VM_145_57_tlinux ~]# ps -ef |grep mysql
root              : ?        :: /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql/var --socket=/tmp/mysql.sock --pid-file=/data/mysql/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql         : ?        :: /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql/var --user=mysql --log-error=/data/mysql/mysqld/mysqld.log --pid-file=/data/mysql/mysqld/mysqld.pid --socket=/tmp/mysql.sock --port=
root         : pts/    :: grep mysql
[root@VM_145_57_tlinux ~]# mysql -uroot -p1234 -S /tmp/mysql.sock
ERROR  (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
[root@VM_145_57_tlinux ~]# ls /tmp/mys^C
[root@VM_145_57_tlinux ~]# /etc/init.d/mysqld restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]
[root@VM_145_57_tlinux ~]# ps -ef | msyql
-bash: msyql: command not found
[root@VM_145_57_tlinux ~]# ps -ef |grep mysql
root              : ?        :: /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql/var --socket=/tmp/mysql.sock --pid-file=/data/mysql/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql         : ?        :: /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql/var --user=mysql --log-error=/data/mysql/mysqld/mysqld.log --pid-file=/data/mysql/mysqld/mysqld.pid --socket=/tmp/mysql.sock --port=
root             : pts/    :: /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql/var --socket=/tmp/mysql.sock --pid-file=/data/mysql/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql        : pts/    :: /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql/var --user=mysql --log-error=/data/mysql/mysqld/mysqld.log --pid-file=/data/mysql/mysqld/mysqld.pid --socket=/tmp/mysql.sock --port=
root         : pts/    :: grep mysql

这下亮了，出现了double进程，没发解释啊。。。【下面这步觉得error】

[root@VM_145_57_tlinux ~]# kill -
[root@VM_145_57_tlinux ~]# kill -
[root@VM_145_57_tlinux ~]# ps -ef |grep mysql
root             : pts/    :: /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql/var --socket=/tmp/mysql.sock --pid-file=/data/mysql/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql        : pts/    :: /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql/var --user=mysql --log-error=/data/mysql/mysqld/mysqld.log --pid-file=/data/mysql/mysqld/mysqld.pid --socket=/tmp/mysql.sock --port=
root         : pts/    :: grep mysql

之前起来的进程被我kill -9之后，发现能连上数据库了，可是innodb存储引擎没起来。

mysql> show engines;
+------------+---------+-----------------------------------------------------------+--------------+------+------------+
| Engine     | Support | Comment                                                   | Transactions | XA   | Savepoints |
+------------+---------+-----------------------------------------------------------+--------------+------+------------+
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                     | NO           | NO   | NO         |
| CSV        | YES     | CSV storage engine                                        | NO           | NO   | NO         |
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance    | NO           | NO   | NO         |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables | NO           | NO   | NO         |
+------------+---------+-----------------------------------------------------------+--------------+------+------------+
 rows in set (0.00 sec)

似乎能看出问题了，进程起来了，socket没有建立，存储引擎没有启动-->innodb正在后台线程操作！！查下errorlog一探究竟：

 :: mysqld_safe Starting mysqld daemon with databases from /data/mysql/var
 ::  InnoDB: Initializing buffer pool, size = .0G
 ::  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn
 ::  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number  

事实上，那个还没创建socket的进程是正在执行double write的回滚工作，继续往下翻页：

InnoDB: Doing recovery: scanned up to log sequence number
 :: mysqld_safe Starting mysqld daemon with databases from /data/mysql/var
 ::  InnoDB: Initializing buffer pool, size = .0G
 ::  InnoDB: Error: cannot allocate  bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB  bytes. Operating system errno:
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most -bit computers the process
InnoDB: memory space is limited to  GB or  GB.
InnoDB: We keep retrying the allocation for  seconds...
InnoDB: Doing recovery: scanned up to log sequence number  

这个是我在innodb引擎正在执行recovery的时候强行启动mysql的报错，提示内存不足。。继续往下翻页：

InnoDB: Doing recovery: scanned up to log sequence number
 ::03InnoDB: Fatal error: cannot allocate the memory for the buffer pool
 :: [ERROR] Plugin 'InnoDB' init function returned error.
 :: [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 :: [Note] Event Scheduler: Loaded  events
 :: [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/tmp/mysql.sock'  port:   Source distribution
InnoDB: Doing recovery: scanned up to log sequence number  3612187648
 [此处省略若干行]
InnoDB: Doing recovery: scanned up to log sequence number
 :: [Note] /usr/libexec/mysqld: Normal shutdown

 :: [Note] Event Scheduler: Purging the queue.  events
 :: [Note] /usr/libexec/mysqld: Shutdown complete

 :: mysqld_safe mysqld from pid file /data/mysql/mysqld/mysqld.pid ended
 :: mysqld_safe Starting mysqld daemon with databases from /data/mysql/var
 ::  InnoDB: Initializing buffer pool, size = .0G
 ::  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 1808 1368235506 [kill -9的结果]
 ::  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number  

看到上面的错误，才恍然大悟，前面的操作是有多危险，要不是mysql的recovery不那么强悍的话，恐怕数据就被我这样弄没了。。。好悬

InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
 ::  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents:
InnoDB: Apply batch completed
 ::  InnoDB: Started; log sequence number
 :: [Note] Event Scheduler: Loaded  events
 :: [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.61'  socket: '/tmp/mysql.sock'  port:   Source distribution

回滚完成，happy启动服务。

下面来讲下innodb_log_file_size在innodb异常时回滚机制：

下面我们参考 http://www.cnblogs.com/zuoxingyu/archive/2012/10/25/2738864.html 大牛的博客，算一下innodb_log_file_size到底多大为最合适：【作为一个粗略的规则，你可以让这个日志足够大到能容纳最多一小时左右的日志】

mysql> pager grep sequence
PAGER set to 'grep sequence'
mysql> show engine innodb status\G select sleep(); show engine innodb status\G
Log sequence number
 row in set (0.00 sec)

 row in set ( min 0.00 sec)

Log sequence number
 row in set (0.00 sec)

(411580730-446445181)/1024/1024/60=1999M，向上取整2G，由于默认有2个日志文件，所以按照当前高峰期计算，设为1G为最佳值。