一次使用innobackupex重新搭建主从复制报错解决方法及注意事项

【环境介绍】

系统环境：CentOS release 6.4 (Final) + Server version: 5.7.18-log MySQL Community Server (GPL) + innobackupex version 2.4.12 Linux (x86_64)

【背景描述】

使用innobackupex重新全备搭建主从复制步骤简单，但是由于历史原因在全备恢复后出现报错：。

[ERROR] InnoDB: Unable to open undo tablespace './/undo001'.
[ERROR] InnoDB: Plugin initialization aborted with error Generic error
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

【实施步骤】

主库操作：

主库备份全库

innobackupex --defaults-file=/etc/my.cnf --user=root --password=mysql -S /tmp/mysql.sock  /root/backup

[root@mysqldb2 backup]# innobackupex --defaults-file=/etc/my.cnf --user=root --password=mysql -S /tmp/mysql.sock  /root/backup
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql/data --server-id=101 --open_files_limit=65535 --tmpdir=/home/mysql/tmp --innodb_buffer_pool_size=1G --innodb_flush_log_at_trx_commit=1 --innodb_undo_tablespaces=3 -

......》》》省略输出内容

xtrabackup: Transaction log of lsn (2458487) to (2458496) was copied.
181222 04:40:18 completed OK!
[root@mysqldb2 backup]#

 

将备份文件拷贝至备库

[root@mysqldb2 backup]# tar -cvf 2018-12-22_04-39-56.tar 2018-12-22_04-39-56/
2018-12-22_04-39-56/
....》》省略部分
2018-12-22_04-39-56/sys/x@0024user_summary_by_statement_type.frm
[root@mysqldb2 backup]# 

[root@mysqldb2 backup]# scp 2018-12-22_04-39-56.tar 192.168.8.100:/root/backup/
root@192.168.8.100's password:
2018-12-22_04-39-56.tar                                                                                                                                    100%  299MB  37.4MB/s   00:08   
[root@mysqldb2 backup]#

备库操作：

关闭数据库

[root@mysqldb1 mysql]# mysqladmin --defaults-file=/etc/my.cnf --protocol=tcp -P3306 shutdown -uroot -pmysql
[root@mysqldb1 mysql]#
[1]+  Done                    mysqld --defaults-file=/etc/my.cnf --user=mysql  (wd: ~)
(wd now: /var/lib/mysql)
[root@mysqldb1 mysql]#
备份原库数据
[root@mysqldb1 mysql]# ls -trl
total 56
drwxr-xr-x.  6 mysql mysql  4096 Dec 22 08:47 data
[root@mysqldb1 mysql]# mv data data_bak

进行全库恢复

进行--apply-log操作

[root@mysqldb1 mysql]# innobackupex --defaults-file=/etc/my.cnf  --apply-log /root/backup/2018-12-22_04-39-56
xtrabackup: recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:256M:autoextend --innodb_log_files_in_group=2 --

....》》省略部分

InnoDB: Shutdown completed; log sequence number 2459176
181222 08:52:03 completed OK!
[root@mysqldb1 mysql]#

进行--copy-back操作

[root@mysqldb1 mysql]# innobackupex --defaults-file=/etc/my.cnf  --copy-back /root/backup/2018-12-22_04-39-56
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql/data --server-id=100 --open_files_limit=65535 --tmpdir=/home/mysql/tmp --innodb_buffer_pool_size=1G --innodb_flush_log_at_trx_commit=1 --innodb_log_buffer_size=32M -

....》》省略部分

181222 08:53:24 [01]        ...done
181222 08:53:24 completed OK!
[root@mysqldb1 mysql]#

修改数据文件目录权限

[root@mysqldb1 mysql]# chown -R mysql:mysql   data

直接启动数据库

[root@mysqldb1 mysql]#mysqld --defaults-file=/etc/my.cnf --user=mysql &

查看日志出现报错：

2018-12-22T13:56:08.963837Z 0 [Warning] option 'table_definition_cache': unsigned value 64 adjusted to 400
2018-12-22T13:56:09.138266Z 0 [Warning] option 'max_binlog_size': unsigned value 2147483648 adjusted to 1073741824
2018-12-22T13:56:09.138392Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-12-22T13:56:09.138450Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2018-12-22T13:56:09.138502Z 0 [Note] mysqld (mysqld 5.7.18-log) starting as process 4873 ...
2018-12-22T13:56:09.197459Z 0 [Note] InnoDB: PUNCH HOLE support not available
2018-12-22T13:56:09.197501Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-12-22T13:56:09.197513Z 0 [Note] InnoDB: Uses event mutexes
2018-12-22T13:56:09.197525Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
2018-12-22T13:56:09.197536Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-12-22T13:56:09.197547Z 0 [Note] InnoDB: Using Linux native AIO
2018-12-22T13:56:09.198473Z 0 [Note] InnoDB: Number of pools: 1
2018-12-22T13:56:09.198675Z 0 [Note] InnoDB: Using CPU crc32 instructions
2018-12-22T13:56:09.200901Z 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 1, chunk size = 128M
2018-12-22T13:56:09.400867Z 0 [Note] InnoDB: Completed initialization of buffer pool
2018-12-22T13:56:09.408553Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-12-22T13:56:09.494523Z 0 [ERROR] InnoDB: Unable to open undo tablespace './/undo001'.
2018-12-22T13:56:09.494551Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2018-12-22T13:56:10.101405Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2018-12-22T13:56:10.101433Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-12-22T13:56:10.101443Z 0 [ERROR] Failed to initialize plugins.
2018-12-22T13:56:10.101451Z 0 [ERROR] Aborting

2018-12-22T13:56:10.101465Z 0 [Note] Binlog end
2018-12-22T13:56:10.101521Z 0 [Note] Shutting down plugin 'CSV'
2018-12-22T13:56:10.102134Z 0 [Note] mysqld: Shutdown complete

 

【问题解决】

从以上报错发现无法打开undo tablespace undo001，查看备库数据目录查看，确实不存在undo001，查看主库存在undo001表空间。

从恢复日志看--apply-log的时候石有undo表空间的信息

InnoDB: Opened 3 undo tablespaces
InnoDB: 3 undo tablespaces made active

从恢复日志看--copy-back的时候并没有拷贝undo表空间的信息。思考为啥没有copy的undo表空间到数据目录信息

innobackupex --defaults-file=/etc/my.cnf  --apply-log /root/backup/2018-12-22_04-39-56

根据思路从命令恢复的可以看出涉及两个文件确认undo表空间信息/etc/my.cnf，/root/backup/2018-12-22_04-39-56

[root@mysqldb1 ~]# cat /etc/my.cnf |grep undo
[root@mysqldb1 ~]# ls -trl /root/backup/2018-12-22_04-39-56|grep undo
-rw-r-----. 1 root root  10485760 Dec 22 08:51 undo003
-rw-r-----. 1 root root  10485760 Dec 22 08:51 undo002
-rw-r-----. 1 root root  10485760 Dec 22 08:51 undo001
[root@mysqldb1 ~]#

从上面的结果可以看出备份文件中是存在undo表空间且是3个undo表空间，但是从参数文件中不存在undo表空间的信息

至此可以初步断定为参数文件中没有指定undo表空间信息，于是在配置文件中添加innodb_undo_tablespaces=3，再次--apply-log操作

观察日志信息，已经存在undo拷贝操作

innobackupex version 2.4.12 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 170eb8c)
181223 03:09:12 [01] Copying undo001 to /var/lib/mysql/data/undo001
181223 03:09:12 [01]        ...done
181223 03:09:12 [01] Copying undo002 to /var/lib/mysql/data/undo002
181223 03:09:13 [01]        ...done
181223 03:09:13 [01] Copying undo003 to /var/lib/mysql/data/undo003
181223 03:09:13 [01]        ...done
181223 03:09:14 [01] Copying ib_logfile0 to /var/lib/mysql/data/ib_logfile0
181223 03:09:15 [01]        ...done
181223 03:09:15 [01] Copying ib_logfile1 to /var/lib/mysql/data/ib_logfile1

于是手工再次启动数据库，数据库正常拉起，没有undo报错，继续搭建主从复制。

【错误日志warring】

2018-12-22T13:56:08.963837Z 0 [Warning] option 'table_definition_cache': unsigned value 64 adjusted to 400
2018-12-22T13:56:09.138266Z 0 [Warning] option 'max_binlog_size': unsigned value 2147483648 adjusted to 1073741824
2018-12-22T13:56:09.138392Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-12-22T13:56:09.138450Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.

根据5.7官方文档对应参数调整如下：

table_definition_cache默认值为400，可以调整400 + (table_open_cache / 2)

max_binlog_size=1024M

explicit_defaults_for_timestamp = ON

secure_file_priv =  " "

【主从复制搭建】

[root@mysqldb1 2018-12-22_04-39-56]# cat xtrabackup_binlog_info
binlog.000011   489     2478c036-bd7a-11e8-85df-080027206a62:5-6,fbfd9bcb-0437-11e9-947d-080027206a62:1-9
[root@mysqldb1 2018-12-22_04-39-56]#

mysql> reset master;
Query OK, 0 rows affected (0.12 sec)

mysql> set global gtid_purged='2478c036-bd7a-11e8-85df-080027206a62:5-6,fbfd9bcb-0437-11e9-947d-080027206a62:1-9';
Query OK, 0 rows affected (0.00 sec)

mysql> change master to master_user= 'repl',master_host='192.168.8.101',master_password='repl',master_port=3306,MASTER_AUTO_POSITION=1;
Query OK, 0 rows affected, 2 warnings (0.48 sec)

mysql> start slave;
Query OK, 0 rows affected (0.04 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.8.101
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: binlog.000013
          Read_Master_Log_Pos: 234
               Relay_Log_File: mysqldb1-relay-bin.000002
                Relay_Log_Pos: 361
        Relay_Master_Log_File: binlog.000013
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

........省略部分输出内容

            Executed_Gtid_Set: 2478c036-bd7a-11e8-85df-080027206a62:5-6,fbfd9bcb-0437-11e9-947d-080027206a62:1-9
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

mysql>

测试数据同步正常

【总结】

innobackupex备份恢复前确认主库跟恢复库的参数一致，可以使用diff命令查看参数的必要性；

可能由于历史原因参数没有写到配置文件报错，应该做好工程修改记录对于问题查找有很大的作用。