Mysql备份 -----innobackupex

一，原理及介绍
〇 xtrabackup能做哪些
    对InnoDB引擎的表做热备
    增量备份
    流压缩传输到另外的服务器上
    在线移动表
    更简单的创建从库
    备份时不增加服务器负载

〇 原理
     备份及恢复大致涉及三个步骤：备份 -> prepare -> 恢复
     备份运行时，工具会记住当时的LSN号，并打开xtrabackup_logfile，然后开始对datafile进行copy，即ibdata1及ibd文件。
     复制需要一定的时间，在复制期间，如果文件被修改，工具将监视redo log file并将每一次更变记录下来，保存在xtrabackup_logfile中。
     接下来处理非事务表如MyISAM的备份操作，innobackupex通过FLUSH TABLES WITH READ LOCK来阻塞DML。
     并在此时获取binlog的position[和GTID]（此处我理解为和mysqldump --single-transaction处理方式类似）
     在做完非事务表的copy之后，执行UNLOCK TABLES，完成备份，并停止记录xtrabackup_logfile。
     接下来就是需要做prepare的过程，该过程类似InnoDB的crash-recovery。
     对redo log进行前滚（到数据文件），并将没提交的事务进行回滚操作（rollback），这样便可以保证数据的一致性，所以对于事务表，整个过程是不会影响写操作的。
 
     注：InnoDB、XtraDB、MyISAM是肯定支持的，其他的存储引擎没有测过。
 
 〇 权限需求
         对datadir需要有rwx的权限。
     MySQL：
         最小所需要的权限有：
         RELOAD
         LOCK TABLES（如果加上--no-lock的话可以不要）
         REPLICATION CLIENT（为了获得binary log的position）
         PROCESS（为了执行show engine innodb status，并且需要查看所有运行的线程）
         其他可能需要用到的权限：
         CREATE TABLESPACE（如果需要通过5.6+ 的TTS恢复/迁移单个表的话）
         SUPER（可能需要在复制环境里启动或者停止slave线程）
         CREATE\INSERT\SELECT（对PERCONA_SCHEMA.xtrabackup_history进行操作）

二、下载安装
官方地址：https://www.percona.com/downloads/XtraBackup/LATEST/

安装：
tar xf Percona-XtraBackup-2.4.12-r170eb8c-el7-x86_64-bundle.tar
rpm -vih percona-xtrabackup*.rpm 或者yum install -y percona-xtrabackup-*

三，全量备份与恢复

•备份数据
语法：innobackupex [--defaults-file=/etc/my.cnf] --user=DBUSER --password=DBUSERPASS [--socket=/var/lib/mysql/mysql.sock --port=3306] /path/to/BACKUP-DIR/

[root@Server data]# innobackupex --defaults-file=/etc/my.cnf --user=root --password= /data/
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --log_bin=server-bin.log --server-id=
xtrabackup: recognized client arguments: --datadir=/var/lib/mysql --log_bin=server-bin.log --server-id=
 :: innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

 ::  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root'  (using passw
ord: YES). ::  version_check Connected to MySQL server
 ::  version_check Executing a version check against the server...
 ::  version_check Done.
 :: Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set
Using server version 5.6.
innobackupex version 2.4. based on MySQL server 5.7. Linux (x86_64) (revision id: 170eb8c)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested , set to
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group =
xtrabackup:   innodb_log_file_size =
InnoDB: Number of pools:
 :: >> log scanned up to ()
xtrabackup: Generating a list of tablespaces
InnoDB: Allocated tablespace ID  for mysql/slave_master_info, old maximum was
 :: [] Copying ./ibdata1 to /data/--29_14--/ibdata1
 :: >> log scanned up to ()
 :: >> log scanned up to ()
 :: >> log scanned up to ()
 :: >> log scanned up to ()
 :: >> log scanned up to ()
 :: >> log scanned up to ()
 :: >> log scanned up to ()
 :: >> log scanned up to ()
.........
 :: Finished backing up non-InnoDB tables and files
 :: Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
xtrabackup: The latest check point (for incremental): ''
xtrabackup: Stopping log copying thread.
. :: >> log scanned up to ()

 :: Executing UNLOCK TABLES
 :: All tables unlocked
 :: Backup created in directory '/data/2018-10-29_14-41-49/'
 :: [] Writing /data/--29_14--/backup-my.cnf
 :: []        ...done
 :: [] Writing /data/--29_14--/xtrabackup_info
 :: []        ...done
xtrabackup: Transaction log of lsn () to () was copied.
 :: completed OK!

当出现innobackupex: completed OK!
出现上面的信息，表示备份已经ok。

查看备份的结果

[root@Server data]# ll
total
drwxr-x---  root root  Oct  : --29_14--
[root@Server data]# cd --29_14--/
[root@Server --29_14--]# ls
backup-my.cnf  ibdata1  mysql  performance_schema  test  test1  xtrabackup_checkpoints  xtrabackup_info  xtrabackup_logfile  zabbix

将数据库备份到远程目录
前提条件：
a.远程机器的目录要存在，同时保证是空目录。远程机器目录非空，备份时报以下错误"xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)"
b.ssh要能连接：使用ssh秘钥免密码输入
innobackupex --user=USER --password=PASSWD  --socket=/var/lib/mysql/mysql.sock --port=3306 --stream=tar /data/temp/  | gzip | ssh root@远程主机host "cat -> /data/temp/mysql_full_backup_2018-10-29.tar.gz"

在备份的文件夹中，有几个文件值得注意：
     xtrabackup_binlog_pos_innodb记录了binlog的position，若开启了GTID，也会将GTID取出。
     在用于备份+binlog恢复或建立slave的场景里十分有用。
     xtrabackup_checkpoints记录了此次备份的类型和lsn号的起始值，是否压缩等
     xtrabackup_info则记录了备份工具的信息，时间，备份对象（是针对全实例还是某库表），是否是增量，binlog位置等
backup-my.cnf文件，则记录了备份时可能涉及到的选项参数，比如系统表空间信息，独立undo表空间信息，redo-log信息等

• 恢复数据
注意：恢复之前
1）要先关闭数据库
2）要删除数据文件和日志文件（也可以mv移到别的地方，只要确保清空mysql数据存放目录就行）
apply-log 类似innodb的crash recovery

[root@Server data]# innobackupex --apply-log /data/--29_14--/ #prepare过程
xtrabackup: recognized server arguments: --innodb_checksum_algorithm=innodb --innodb_log_checksum_algorithm=innodb --innodb_data_file_path=
ibdata1:12M:autoextend --innodb_log_files_in_group= --innodb_log_file_size= --innodb_fast_checksum= --innodb_page_size= --innodb_log_block_size= --innodb_undo_directory=. --innodb_undo_tablespaces= --server-id= --redo-log-version= xtrabackup: recognized client arguments: --innodb_checksum_algorithm=innodb --innodb_log_checksum_algorithm=innodb --innodb_data_file_path=
ibdata1:12M:autoextend --innodb_log_files_in_group= --innodb_log_file_size= --innodb_fast_checksum= --innodb_page_size= --innodb_log_block_size= --innodb_undo_directory=. --innodb_undo_tablespaces= --server-id= --redo-log-version=  :: innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

innobackupex version 2.4. based on MySQL server 5.7. Linux (x86_64) (revision id: 170eb8c)
xtrabackup: cd to /data/--29_14--/
xtrabackup: This target seems to be not prepared yet.
InnoDB: Number of pools:
xtrabackup: xtrabackup_logfile detected: size=, start_lsn=()
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = .
xtrabackup:   innodb_log_files_in_group =
xtrabackup:   innodb_log_file_size =
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = .
xtrabackup:   innodb_log_files_in_group =
xtrabackup:   innodb_log_file_size =
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using  bytes for buffer pool (set by --use-memory parameter)
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.
InnoDB: Number of pools:
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn
InnoDB: Doing recovery: scanned up to log sequence number  (%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent:
                                                            InnoDB: Apply batch completed
InnoDB: xtrabackup: Last MySQL binlog file position , file name ./server-bin.
InnoDB: Creating shared tablespace for temporary tables
InnoDB: Setting file './ibtmp1' size to  MB. Physically writing the file full; Please wait ...
InnoDB: File './ibtmp1' size is now  MB.
InnoDB:  redo rollback segment(s) found.  redo rollback segment(s) are active.
InnoDB:  non-redo rollback segment(s) are active.
InnoDB: Waiting for purge to start
InnoDB: 5.7. started; log sequence number
InnoDB: xtrabackup: Last MySQL binlog file position , file name ./server-bin.

xtrabackup: starting shutdown with innodb_fast_shutdown =
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number
InnoDB: Number of pools:
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = .
xtrabackup:   innodb_log_files_in_group =
xtrabackup:   innodb_log_file_size =
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.
InnoDB: Number of pools:
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -
InnoDB: Setting log file ./ib_logfile101 size to  MB
InnoDB: Setting log file ./ib_logfile1 size to  MB
InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
InnoDB: New log files created, LSN=
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn
InnoDB: Doing recovery: scanned up to log sequence number  (%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: xtrabackup: Last MySQL binlog file position , file name ./server-bin.
InnoDB: Removed temporary tablespace data file: "ibtmp1"
InnoDB: Creating shared tablespace for temporary tables
InnoDB: Setting file './ibtmp1' size to  MB. Physically writing the file full; Please wait ...
InnoDB: File './ibtmp1' size is now  MB.
InnoDB:  redo rollback segment(s) found.  redo rollback segment(s) are active.
InnoDB:  non-redo rollback segment(s) are active.
InnoDB: Waiting for purge to start
InnoDB: 5.7. started; log sequence number
xtrabackup: starting shutdown with innodb_fast_shutdown =
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number
 :: completed OK!

当出现innobackupex: completed OK!
出现上面的信息，表示恢复已经ok。

# cp -r 2018-10-29_14-41-49/*  /var/lib/mysql/      将数据拷贝到数据库的数据目录下
也可以用此命令拷贝  innobackupex --copy-back /var/lib/mysql/
# chown mysql:mysql /var/lib/mysql/ -R
重启数据库

四，增量备份与恢复
①增备方法与全备不一样：
     innobackupex --user= --password= --incremental $new_dir --incremental-basedir=$basedir
 
    其中--incremental是本次增量备份存放目录
     $new_dir是表示将增量备份出来的东西放在哪个目录
     --incremental-basedir则表示，针对哪一次备份做增量备份
 
     备份的差异在目录的xtrabackup_checkpoints中查看：
     比如：
     $basedir中内容： 
     backup_type = full-prepared
     from_lsn = 0
     to_lsn = 2048706153
     last_lsn = 2048706153
     compact = 0
     recover_binlog_info = 0 
 
     $new_bkdir中内容：
     backup_type = incremental
     from_lsn = 2048706153
     to_lsn = 2048876543
     last_lsn = 2048876543
     compact = 0
     recover_binlog_info = 0
  
     可以注意一下增备的from_lsn号
     大于这个LSN号的页都是被变更过的，这些偏移量，也就是需要被增量备份出去的
 
 ②prepare：
     prepare过程：
     从第一个备份开始（也就是全量）做prepare，再将往后的增量备份依次添加到全量备份中。
     注意，此处多了一个参数即--redo-only，该参数是指将已提交的事务应用，未提交的事务回滚。
     此外，--incremental-dir也是在之前没有用到过的，这个参数代表需要被合并进去的增量备份目录。
     注意，此处多次的增量备份是指：针对上次的增量备份做的增量。
    
     也就是可以理解为：
         全备：500GB
         第一次增量备份：2GB
         第二次增量备份：1GB（针对第一次增量备份的增量数据）
         ……
         第n次
 
     按照备份顺序做prepare，也就是prepare的顺序为：
     第一次全备 -> 增量备份1 -> 增量备份2 -> ... -> 增量备份n
     第一次全备的prepare：innobackup --apply-log --redo-only $basedir
     第二次prepare：innobackup --apply-log --redo-only $basedir --incremental-dir=$new_dir_1（此处的$new_dir_1也就是第一次增量备份）
     ......
     第n次prepare：innobackup --apply-log $basedir --incremental-dir=$new_dir_n（此处的$new_dir_n也就是最近也就是最后一次的增量备份
     最后一次增量备份的prepare，不需要指定--redo-only
 
     最后将增量备份和全备进行合并，将未提交的事务回滚，这个操作和全量prepare无异：
     innobackup --apply-log $basedir

③恢复到datadir：
     和全量无异，直接copyback就行了
     innobackupex --copy-back $basedir
 
  增量备份的prepare有点特殊，还是小结一下：
      ① prepare完备（加上--redo-only）
      ② prepare每一次增量备份到完备中，需要加上--redo-only，最后一次增量备份的prepare不需要加--redo-only
      ③ 对生成的最终完备做--apply-log

执行第一次增量备份之后，可以再做一次增量备份
此时有两种增量备份方法：
     第一种，总是针对basedir做增量，这个方式恢复起来就特别简单了，只需要将最后一次的增量备份合并到全量备份里，就可以恢复了。
     第二种，总是针对上一次的增量，做增量备份。这个方式的恢复，就要逐一合并了，也就是我上述所说看起来有点复杂的增备思路。

图示：
第一种：
     总是将1月1日的全备作为basedir，所以FROM_LSN号总是5000。

第二种：
     总是把上一次（最近一次）的备份作为basedir。