KingbaseES R3 集群主库归档失败案例

案例说明：

本案例用于KingbaseES R3集群归档进程归档日志失败的处理，对于一线的生产环境具有一定的参考意义。

数据库版本：

TEST=# select version();

                                                         VERSION

---------------------------------------------------------------------------------------------------------------

 Kingbase V008R003C002B0270 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit

(1 row)

集群架构：

一、案例故障现象（主库归档失败）

1、主库数据库服务进程

[kingbase@node1 bin]$ ps -ef |grep kingbase

kingbase  8180     1  0 13:44 ?        00:00:00 /home/kingbase/cluster/kha/db/bin/kingbase -D /home/kingbase/cluster/kha/db/data

kingbase  8181  8180  0 13:44 ?        00:00:00 kingbase: logger process

kingbase  8183  8180  0 13:44 ?        00:00:00 kingbase: checkpointer process

kingbase  8184  8180  0 13:44 ?        00:00:00 kingbase: writer process

kingbase  8185  8180  0 13:44 ?        00:00:00 kingbase: wal writer process

kingbase  8186  8180  0 13:44 ?        00:00:00 kingbase: autovacuum launcher process

kingbase  8187  8180  0 13:44 ?        00:00:00 kingbase: archiver process   failed on 000000020000000000000010

kingbase  8188  8180  0 13:44 ?        00:00:00 kingbase: stats collector process

kingbase  8189  8180  0 13:44 ?        00:00:00 kingbase: bgworker: syslogical supervisor

kingbase  8253  8180  0 13:44 ?        00:00:00 kingbase: wal sender process SYSTEM 192.168.7.243(51049) streaming 0/160001B0

2、查看sys_log日志

2021-03-01 13:49:30.693 CST,,,8187,,603c7f31.1ffb,23,,2021-03-01 13:44:17 CST,,0,LOG,00000,"archive command failed with exit code 45","The failed archive command was: /home/kingbase/cluster/kha/db/bin/sys_rman_v6 --config /home/kingbase/kbbr3_repo/sys_rman_v6.conf --stanza=kingbase archive-push sys_xlog/000000020000000000000010",,,,,,,,""

2021-03-01 13:49:30.694 CST,,,8187,,603c7f31.1ffb,24,,2021-03-01 13:44:17 CST,,0,WARNING,01000,"archiving transaction log file ""000000020000000000000010"" failed too many times, will try again later",,,,,,,,,""

二、故障处理步骤

1、查看archive日志归档配置

2、查看归档配置文件和目录信息

[kingbase@node1 sys_log]$ ls -lh /home/kingbase/kbbr3_repo/sys_rman_v6.conf

-rw-rw-r-- 1 kingbase kingbase 589 Mar  1 12:26 /home/kingbase/kbbr3_repo/sys_rman_v6.conf

[kingbase@node1 sys_log]$ cat /home/kingbase/kbbr3_repo/sys_rman_v6.conf

# Genarate by script at 20210301122559, should not change manually

[kingbase]

kb1-path=/home/kingbase/cluster/kha/db/data

kb1-port=54321

kb1-user=SUPERMANAGER_V8ADMIN

kb1-pass=S0lOR0JBU0VBRE1JTg==

kb2-path=/home/kingbase/cluster/kha/db/data

kb2-port=54321

kb2-user=SUPERMANAGER_V8ADMIN

kb2-pass=S0lOR0JBU0VBRE1JTg==

kb2-host=192.168.7.243

kb2-host-user=kingbase

[global]

repo1-path=/home/kingbase/kbbr3_repo

repo1-retention-full=5

log-path=/tmp/

log-level-file=info

log-level-console=info

log-subprocess=y

process-max=4

#### default gz, support: gz none

compress-type=gz

compress-level=3

3、执行手工归档

[kingbase@node1 bin]$ /home/kingbase/cluster/kha/db/bin/sys_rman_v6 --config /home/kingbase/kbbr3_repo/sys_rman_v6.conf --stanza=kingbase archive-push /home/kingbase/cluster/kha/db/data/sys_xlog/000000020000000000000010

2021-03-01 14:43:20.928 P00   INFO: archive-push command begin 2.27: [/home/kingbase/cluster/kha/db/data/sys_xlog/000000020000000000000010] --compress-level=3 --compress-type=gz --config=/home/kingbase/kbbr3_repo/sys_rman_v6.conf --log-level-console=info --log-level-file=info --log-path=/tmp --log-subprocess --kb2-host=192.168.7.243 --kb1-path=/home/kingbase/cluster/kha/db/data --kb2-path=/home/kingbase/cluster/kha/db/data --process-max=4 --repo1-path=/home/kingbase/kbbr3_repo --stanza=kingbase

2021-03-01 14:43:21.203 P00   INFO: pushed WAL file '000000020000000000000010' to the archive

2021-03-01 14:43:21.204 P00   INFO: archive-push command end: completed successfully (276ms)

# 查看归档日志文件

[kingbase@node1 data]$ ls -lh /home/kingbase/kbbr3_repo/archive/kingbase/9.6-1/0000000200000000

total 208K

-rw-r----- 1 kingbase kingbase 91K Mar  1 12:20 00000002000000000000000E-583ac46b5270f365463cb0bfb3b96185af6492dd.gz

-rw-r----- 1 kingbase kingbase 303 Mar  1 12:20 00000002000000000000000F.00000028.backup

-rw-r----- 1 kingbase kingbase 83K Mar  1 12:20 00000002000000000000000F-1c450ca422ee6312e8a69dcb7d8c446a99425995.gz

-rw-r----- 1 kingbase kingbase 28K Mar  1 12:21 000000020000000000000010-044ab3927144a6510a42ce9c2bc331cf209aff56.gz

=从以上信息可知，手工归档成功，可以说明归档的配置文件及目录权限等没有问题。=

4、重启集群测试

[kingbase@node1 bin]$ ps -ef |grep kingbase

kingbase  8180     1  0 13:44 ?        00:00:00 /home/kingbase/cluster/kha/db/bin/kingbase -D /home/kingbase/cluster/kha/db/data

kingbase  8181  8180  0 13:44 ?        00:00:00 kingbase: logger process

kingbase  8183  8180  0 13:44 ?        00:00:00 kingbase: checkpointer process

kingbase  8184  8180  0 13:44 ?        00:00:00 kingbase: writer process

kingbase  8185  8180  0 13:44 ?        00:00:00 kingbase: wal writer process

kingbase  8186  8180  0 13:44 ?        00:00:00 kingbase: autovacuum launcher process

kingbase  8187  8180  0 13:44 ?        00:00:00 kingbase: archiver process   failed on 000000020000000000000010

kingbase  8188  8180  0 13:44 ?        00:00:00 kingbase: stats collector process

kingbase  8189  8180  0 13:44 ?        00:00:00 kingbase: bgworker: syslogical supervisor

kingbase  8253  8180  0 13:44 ?        00:00:00 kingbase: wal sender process SYSTEM 192.168.7.243(51049) streaming 0/160001B0

=== 从以上信息可知，归档仍然失败。===

5、修改archive_command配置（跳过归档）

6、重启集群测试

[kingbase@node1 bin]$ ps -ef |grep kingbase

kingbase 21906  5688  0 14:09 pts/0    00:00:00 ./ksql -U SYSTEM -W ******** TEST

kingbase 23167     1  0 14:11 ?        00:00:00 /home/kingbase/cluster/kha/db/bin/kingbase -D /home/kingbase/cluster/kha/db/data

kingbase 23168 23167  0 14:11 ?        00:00:00 kingbase: logger process

kingbase 23170 23167  0 14:11 ?        00:00:00 kingbase: checkpointer process

kingbase 23171 23167  0 14:11 ?        00:00:00 kingbase: writer process

kingbase 23172 23167  0 14:11 ?        00:00:00 kingbase: wal writer process

kingbase 23173 23167  0 14:11 ?        00:00:00 kingbase: autovacuum launcher process

kingbase 23174 23167  0 14:11 ?        00:00:00 kingbase: archiver process

kingbase 23175 23167  0 14:11 ?        00:00:00 kingbase: stats collector process

kingbase 23176 23167  0 14:11 ?        00:00:00 kingbase: bgworker: syslogical supervisor

kingbase 23194 23167  0 14:11 ?        00:00:00 kingbase: wal sender process SYSTEM 192.168.7.243(54037) streaming 0/180000D0

=== 从以上信息可知，已经没有归档失败的状态信息。===

7、在主库手工执行wal日志切换

TEST=# select sys_switch_xlog();

 SYS_SWITCH_XLOG

-----------------

 0/180000E8

(1 row)

TEST=# select sys_switch_xlog();

 SYS_SWITCH_XLOG

-----------------

 0/19000078

(1 row)

TEST=# select sys_switch_xlog();

 SYS_SWITCH_XLOG

-----------------

 0/1A000000

(1 row)

8、再重新恢复archive_command配置

9、重启集群测试

[kingbase@node1 bin]$ ps -ef |grep kingbase

kingbase 25979     1  0 14:15 ?        00:00:00 /home/kingbase/cluster/kha/db/bin/kingbase -D /home/kingbase/cluster/kha/db/data

kingbase 25980 25979  0 14:15 ?        00:00:00 kingbase: logger process

kingbase 25983 25979  0 14:15 ?        00:00:00 kingbase: checkpointer process

kingbase 25984 25979  0 14:15 ?        00:00:00 kingbase: writer process

kingbase 25985 25979  0 14:15 ?        00:00:00 kingbase: wal writer process

kingbase 25986 25979  0 14:15 ?        00:00:00 kingbase: autovacuum launcher process

kingbase 25987 25979  0 14:15 ?        00:00:00 kingbase: archiver process

kingbase 25988 25979  0 14:15 ?        00:00:00 kingbase: stats collector process

kingbase 25989 25979  0 14:15 ?        00:00:00 kingbase: bgworker: syslogical supervisor

kingbase 26006 25979  0 14:15 ?        00:00:00 kingbase: wal sender process SYSTEM 192.168.7.243(54457) streaming 0/1B0000D0

=== 从以上信息可知，已经没有归档失败的状态信息。===

10、测试wal日志归档

1）执行wal日志切换（主库）

TEST=# select sys_switch_xlog();

 SYS_SWITCH_XLOG

-----------------

 0/1C000078

(1 row)

TEST=# select sys_switch_xlog();

 SYS_SWITCH_XLOG

-----------------

 0/1E000000

(1 row)

2）查看归档信息

[kingbase@node1 0000000200000000]$ pwd

/home/kingbase/kbbr3_repo/archive/kingbase/9.6-1/0000000200000000

[kingbase@node1 0000000200000000]$ ls -lh

total 460K

-rw-r----- 1 kingbase kingbase 91K Mar  1 12:20 00000002000000000000000E-583ac46b5270f365463cb0bfb3b96185af6492dd.gz

-rw-r----- 1 kingbase kingbase 303 Mar  1 12:20 00000002000000000000000F.00000028.backup

-rw-r----- 1 kingbase kingbase 83K Mar  1 12:20 00000002000000000000000F-1c450ca422ee6312e8a69dcb7d8c446a99425995.gz

-rw-r----- 1 kingbase kingbase 28K Mar  1 12:21 000000020000000000000010-044ab3927144a6510a42ce9c2bc331cf209aff56.gz

-rw-r----- 1 kingbase kingbase 83K Mar  1 14:22 00000002000000000000001B-f64eadf9b3ecb50ce6925cdc8c196bf33af4cc8c.gz

-rw-r----- 1 kingbase kingbase 84K Mar  1 14:22 00000002000000000000001C-300ed282be050b1194cc15b974b5b11b120d2076.gz

-rw-r----- 1 kingbase kingbase 84K Mar  1 14:23 00000002000000000000001D-249ab2fafe02a51ca06846098d4bad2f786d1422.gz

=从归档日志信息看，在wal日志发生切换时，产生了归档；但是前面在修改archive_command='/bin/true'参数后，导致一部分wal日志没有归档。==

11、查看数据库服务进程

[kingbase@node1 0000000200000000]$ ps -ef |grep kingbase

.......

kingbase 25979     1  0 14:15 ?        00:00:00 /home/kingbase/cluster/kha/db/bin/kingbase -D /home/kingbase/cluster/kha/db/data

kingbase 25980 25979  0 14:15 ?        00:00:00 kingbase: logger process

kingbase 25983 25979  0 14:15 ?        00:00:00 kingbase: checkpointer process

kingbase 25984 25979  0 14:15 ?        00:00:00 kingbase: writer process

kingbase 25985 25979  0 14:15 ?        00:00:00 kingbase: wal writer process

kingbase 25986 25979  0 14:15 ?        00:00:00 kingbase: autovacuum launcher process

kingbase 25987 25979  0 14:15 ?        00:00:00 kingbase: archiver process   last was 00000002000000000000001D

kingbase 25988 25979  0 14:15 ?        00:00:00 kingbase: stats collector process

kingbase 25989 25979  0 14:15 ?        00:00:00 kingbase: bgworker: syslogical supervisor

kingbase 26006 25979  0 14:15 ?        00:00:00 kingbase: wal sender process SYSTEM 192.168.7.243(54457) streaming 0/1E000060

=== 从以上信息获知，数据库归档进程正常。===

三、总结

此案例解决了KingbaseES R3集群归档失败的故障，对于数据库归档失败在原生PostgreSQL也发现过此类问题，处理起来比较繁琐，主库归档恢复正常后，需要对数据库做一次物理的全备，因为中间缺失了一部分wal日志的归档。

KingbaseES R3 集群主库归档失败案例的更多相关文章

KingbaseES R3 集群一键修改集群用户密码案例
案例说明: 在KingbaseES R3集群的最新版本中增加了kingbase_monitor.sh一键修改集群用户密码的功能,本案例是对此功能的测试. kingbaseES R3集群一键修改密码说明 ...
kingbaseES R3 集群修改data路径测试案例
案例说明: 默认KingbaseES R3集群部署后,数据存储目录(data)在/home/kingbase下,部署时不能更改:本案例是在部署完成后,迁移data目录到其他指定的存储位置. 数据库版本 ...
KingbaseES R3 集群cluster日志切割和清理案例
案例说明: 对于KingbaseES R3集群的cluster日志默认系统是不做切割和清理的,随着运行时长的增加,日志将增长为一个非常大的文件,占用比较大的磁盘空间,并且在分析问题读取大文件时效率很低 ...
KingbaseES R3集群在线删除数据节点案例
案例说明: kingbaseES R3集群一主多从的架构,一般有两个节点是集群的管理节点,所有的节点都可以为数据节点:对于非管理节点的数据节点可以在线删除:但是对于管理节点,无法在线删除,如果删除管理 ...
KingbaseES R3 集群删除test库导致主备无法切换问题
案例说明: 在KingbaseES R3集群中,kingbasecluster进程会通过test库访问,连接后台数据库服务测试:如果删除test数据库,导致后台数据库服务访问失败,在集群主备切换时,无 ...
KingbaseES R3 集群修改system用户密码方案
方案说明: 对于kingbaseES R3集群修改system密码相比单机环境有一定的复杂性,需要修改的位置如下: 1)数据库中system用户密码,可以用alter user命令修改 2)在reco ...
KingbaseES R3集群备库执行sys_backup.sh物理备份案例
案例说明: KingbaseES R3的后期版本支持通过sys_backup.sh执行sys_rman的物理备份,实际上是调用了sys_rman_v6的工具做物理备份.本案例是在备库上执行集群的备份, ...
kingbaseES R3 集群备库转换为单实例库案例
案例说明: 在生产环境需要将集群中架构转换为单实例环境,本案例以备库转换为单实例库为案例,介绍了两种方案,一种在数据库数据量小的环境下采用 sys_dumpall 导出导入方式建立单实例库:另外一种是 ...
KingbaseES V8R3集群管理和维护案例之---failover切换wal日志变化分析
案例说明: 本案例通过对KingbaseES V8R3集群failover切换过程进行观察,分析了主备库切换后wal日志的变化,对应用者了解KingbaseES V8R3(R6) failover ...

随机推荐

Java实用类(五) -Math类和指定范围的随机数
1.Math类 java.lang.Math类提供了常用的数学运算方法和两个静态常量E(自然对数的底数) 和PI(圆周率) // 绝对值 System.out.println(Math.abs(-3. ...
小样本利器2.文本对抗+半监督 FGSM & VAT & FGM代码实现
小样本利器2.文本对抗+半监督 FGSM & VAT & FGM代码实现上一章我们聊了聊通过一致性正则的半监督方案,使用大量的未标注样本来提升小样本模型的泛化能力.这一章我们结合FG ...
关于nginx 和 uwsgi
关于nginx和uWSGI和Django之间的关系,我觉得要理一下. 原文链接为什么要用nginx 因为我们要使用https协议访问.(y总说django不支持,但是我查了一下,django也可以支 ...
校验日期格式为yyyy-MM-dd
/** * 校验时间 * * @param text * @return */ public static boolean checkTime(String text) { DateFormat fo ...
11.4 Android Studio如何设置代理
有些网络环境下,Android Studio下载无法下载依赖,这个时候就要配置代理,至于代理的问题,大家要自己解决. 获取代理信息一般要获取如下信息: 地址:可以是域名和IP 端口: 代理类型:常用 ...
如何用Fiddler对APP进行网络测试
什么是Fiddler Fiddler是一个http协议调试代理工具,它能够记录并检查所有你的电脑和互联网之间的http通讯,设置断点,查看所有的"进出"Fiddler的数据(指co ...
scanf读入与printf输出
作为一个资深$cin,cout$玩家,在多次因为$cin$太慢被吊打后,开始反思有必要认真地学一下$scanf$和$printf$了$\cdot \cdot \cdot$ 格式 $scanf( &qu ...
Python进程管理神器——Supervisor
一.简介 Supervisor 是一款 Python 开发的进程管理系统,允许用户监视和控制 Linux 上的进程,能将一个普通命令行进程变为后台守护进程,异常退出时能自动重启详细介绍查阅:Supe ...
如何优化API？8个实用技巧！【eolink翻译】
使用 API 可以让公司利用现代连接的力量来帮助他们扩大全球影响力.传输数据和改进集成.由于 API 使企业能够简化流程并增强可用性,所以企业会使用一些优化策略,不断优化流程,比如接下来要说到的8个技 ...
工作流引擎在vivo营销自动化中的应用实践 | 引擎篇03
作者:vivo 互联网服务器团队- Cheng Wangrong 本文是<vivo营销自动化技术解密>的第4篇文章,分析了在营销自动化业务引入工作流技术的背景和工作流引擎的介绍,同时介绍了 ...

KingbaseES R3 集群主库归档失败案例

KingbaseES R3 集群主库归档失败案例的更多相关文章

随机推荐

热门专题