KingbaseES V8R6单实例外部备份故障案例

案例说明:

在KingbaseES V8R6单实例环境，配置外部备份服务器使用sys_backup.sh物理备份时，出现以下”WAL segment xxx was not archived before the 60000ms timeout“故障。操作步骤见：《KingbaseES V8R6 单实例sys_backup.sh外部备份案例》https://www.cnblogs.com/tiany1224/p/16334032.html

数据库版本信息：

test=# select version();
                                                       version
------------------------------------------------------------------------------------------------------------------
 KingbaseES V008R006C005B0054 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit
(1 row)

• 故障现象：
  
  

• 故障分析：

1、查看数据库节点下sys_log日志

2022-05-30 23:18:57.147 CST [21529] LOG:  archive command failed with exit code 55
2022-05-30 23:18:57.147 CST [21529] DETAIL:  The failed archive command was: export TZ=Asia/Shanghai;/opt/Kingbase/ES/V8R6_054/Server/bin/sys_rman --config /home/kingbase/kbbr1_repo/sys_rman.conf --stanza=kingbase archive-push sys_wal/00000001000000000000001C
2022-05-30 23:18:57.147 CST [21529] WARNING:  archiving write-ahead log file "00000001000000000000001C" failed too many times, will try again later

2、提取日志归档信息手工执行归档

[kingbase@node1 data]$ /opt/Kingbase/ES/V8R6_054/Server/bin/sys_rman --config /home/kingbase/kbbr1_repo/sys_rman.conf --stanza=kingbase archive-push sys_wal/00000001000000000000001C

2022-05-30 23:23:32.207 P00   INFO: archive-push command begin 2.27: [sys_wal/00000001000000000000001C] --config=/home/kingbase/kbbr1_repo/sys_rman.conf --exec-id=31480-4e35a388 --kb1-path=/data/kingbase/v8r6_054/data --log-level-console=info --log-level-file=info --log-path=/opt/Kingbase/ES/V8R6_054/Server/log --log-subprocess --repo1-host=192.168.8.100 --repo1-host-config=/home/kingbase/kbbr1_repo/sys_rman.conf --repo1-host-user=kingbase --repo1-path=/home/kingbase/kbbr1_repo --stanza=kingbase
ERROR: [103]: unable to find a valid repository:
       repo1: [UnknownError] remote-0 process on '192.168.8.100' terminated unexpectedly [127]: bash: /opt/Kingbase/ES/V8R6_054/KESRealPro/V008R006C005B0054/Server/bin/sys_rman: No such file or directory
2022-05-30 23:23:32.461 P00   INFO: archive-push command end: aborted with exception [103]

3、查看备份服务器路径信息

=从备份服务器查询到，原路径信息“/opt/Kingbase/ES/V8R6_054/Server"，由于此版本的数据库软件安装存储路径使用了符号链接(如下图所示），导致在repo节点配置文件路径时，和数据库节点的存储路径不一致，在repo节点读取文件失败。=

数据节点文件存储路径：

• 故障解决：
  
  1、在repo节点上重新创建存储目录

[kingbase@srv01 bin]$ mkdir -p /opt/Kingbase/ES/V8R6_054/KESRealPro/V008R006C005B0054/Server

[kingbase@srv01 V8R6_054]$ cd /opt/Kingbase/ES/V8R6_054/KESRealPro/V008R006C005B0054/Server/
[kingbase@srv01 Server]$ ls -lh
total 338M
drwxr-xr-x 2 kingbase kingbase 4.0K May 31 14:20 bin
-rw------- 1 kingbase kingbase 338M May 31 14:14 db.zip
drwxrwxr-x 5 kingbase kingbase 8.0K Apr  7 16:17 lib
drwx------ 2 kingbase kingbase  100 May 31 14:23 log
drwxrwxr-x 8 kingbase kingbase 4.0K Apr  7 16:17 share

2、编辑sys_backup.conf文件

3、重新执行初始化

4、 重新执行手工归档（归档成功）

[kingbase@node1 data]$ /opt/Kingbase/ES/V8R6_054/Server/bin/sys_rman --config /home/kingbase/kbbr1_repo/sys_rman.conf --stanza=kingbase archive-push sys_wal/00000001000000000000001C

2022-05-30 23:26:17.480 P00   INFO: archive-push command begin 2.27: [sys_wal/00000001000000000000001C] --config=/home/kingbase/kbbr1_repo/sys_rman.conf --exec-id=32254-8f3f05e4 --kb1-path=/data/kingbase/v8r6_054/data --log-level-console=info --log-level-file=info --log-path=/opt/Kingbase/ES/V8R6_054/Server/log --log-subprocess --repo1-host=192.168.8.100 --repo1-host-config=/home/kingbase/kbbr1_repo/sys_rman.conf --repo1-host-user=kingbase --repo1-path=/home/kingbase/kbbr1_repo --stanza=kingbase

WARN: WAL file '00000001000000000000001C' already exists in the repo1 archive with the same checksum
      HINT: this is valid in some recovery scenarios but may also indicate a problem.
2022-05-30 23:26:17.924 P00   INFO: pushed WAL file '00000001000000000000001C' to the archive
2022-05-30 23:26:18.026 P00   INFO: archive-push command end: completed successfully (549ms)

• 总结

  对于sys_backup.sh工具执行物理备份时，很多故障都体现在”归档超时“的问题，解决以上问题，可以使用以下思路：

      1、查看数据节点的sys_log日志，查看是否有归档失败对应的日志信息。
      2、如果有以上日志信息，提取日志信息，通过手工方式执行归档命令，查看执行失败的错误提示。
      3、按照错误提示，查看对应的配置文件，和归档相关的配置是否正确，如存储路径、目录权限、wal日志缺失等，解决归档失败问题。
      4、如在业务量高峰时备份，会产生I/O拥塞，导致归档失败，可以调高archive_timeout参数的阈值，建议不要在业务高峰执行备份操作。