今天遇到了一个客户，问题是这样的，客户构筑了一个RACtoRAC的 LOGICAL STANDBY环境。
并用EM在监视同期情况，发现EM页面上55115和55116这两个SEQUENCE一直在应用。

首先向客户要了下列的资料

　Primary端
Script to Collect Data Guard Primary Site Diagnostic Information for Version 10g and Above (Including RAC). (Doc ID 1577401.1)

　Standby端
　　SRDC - Collect Logical Standby Database Information (Doc ID 1910065.1)

=============================
Logical Standby Database (from all Instances if RAC is used)

ALERT.LOG covering at least the Time since the last Instance Startup and the Problem
       Output from Note: 1577407.1 - Script to Collect Data Guard Logical Standby Diagnostic Information for Version 10g and above (Including RAC)
       Output from Note: 1296954.1 - How to monitor the progress of the logical standby
       TNSNAMES.ORA, LISTENER.ORA
       Latest LSPx- and ASxx-Tracefiles from Background Dump-Directory
=============================

拿到资料后，先看Standby端的DB alert log

1. 是不是Redo没有传到Standby端

[root@standby ~]# grep 'RFS LogMiner: Registered logfile.*thread_1' alert_mykartes.log　★从Standby端抽取传输信息
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55109.346.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55110.347.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55108.351.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55114.354.955743503] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567] to LogMiner session id [2]　★55116之后的数字也出现了
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55117.391.955749633] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55118.393.955749695] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55119.399.955750553] to LogMiner session id [2]　★RFS进程接收REDO看起来很正常
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55120.406.955751745] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55121.415.955753049] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55122.422.955754083] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55123.428.955755081] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55124.439.955756675] to LogMiner session id [2]　★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_27/thread_1_seq_55125.442.955756975] to LogMiner session id [2]

2. 55115和55116，这两个SEQUENCE应用出了什么错么

[root@yan1 ~]# grep 'LOGMINER: Begin mining logfile for session 2 thread 1 sequence 5511' alert_mykartes.log
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55110, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55110.347.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55111, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55112, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55113, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55114, +DG1/mykartes/onlinelog/group_10.315.955743237
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/onlinelog/group_9.314.955743237
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55111, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55112, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55113, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55114, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55114.354.955743503
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421　★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567　★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421　★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567　★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421　★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567　★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421　★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567　★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421　★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567　★55116

我了个去，为啥这两个SEQUENCE会应用这么多次！
看起来好像是LSP进程一直想要应用他们，但就是应用不了。

那就只能从 Doc ID 1910065.1 的资料里找线索了。

Doc ID 1910065.1 的脚本执行结果被客户放在了 new_dg_lsby_diag_mykartes_20170928_1729.html 里面

3.Check 一下REDO的应用状况

new_dg_lsby_diag_mykartes_20170928_1729.html 的部分信息
===========================
SQL> SELECT thread#, sequence#, first_change#, next_change#, dict_begin beg, dict_end end, TO_CHAR(timestamp, 'hh:mi:ss') timestamp, (CASE WHEN l.next_change# < p.read_scn THEN 'YES' WHEN l.first_change# < p.applied_scn THEN 'CURRENT' ELSE 'NO' END) applied FROM dba_logstdby_log l, dba_logstdby_progress p ORDER BY thread#, first_change#;

THREAD# SEQUENCE# FIRST_CHANGE# NEXT_CHANGE# BEG END TIMESTAM APPLIED
1 55107 3734503196 3734503595 NO NO 08:17:43 YES
1 55108 3734503595 3734504891 YES NO 08:17:43 YES
1 55109 3734504891 3734506783 NO YES 08:17:43 YES
1 55110 3734506783 3734506876 NO NO 08:17:43 YES
1 55111 3734506876 3734507595 NO NO 08:17:43 YES
1 55112 3734507595 3734508534 NO NO 08:17:43 YES
1 55113 3734508534 3734509236 NO NO 08:17:43 YES
1 55114 3734509236 3734509452 NO NO 08:18:22 YES
1 55115 3734509452 3734532403 NO NO 08:33:41 CURRENT　★55115在应用中
1 55116 3734532403 3734613007 NO NO 09:09:27 CURRENT　★55116在应用中
1 55117 3734613007 3734706486 NO NO 10:00:32 NO
1 55118 3734706486 3734805339 NO NO 10:01:35 NO
===========================

有戏啊，看来就是这两个家伙的应用hang了。

4.Check 一下REDO应用时有没有什么Error

new_dg_lsby_diag_mykartes_20170928_1729.html により抜粋
===========================
SQL> -- Verify that log apply services on the standby are currently running.　★
SQL> -- If the query against V$LOGSTDBY returns no rows then logical apply is not running.
SQL>
SQL> column status format a50 wrap
SQL> column type format a11
SQL> set numwidth 15
SQL>
SQL> SELECT type, status, high_scn FROM v$logstdby;

TYPE STATUS HIGH_SCN
COORDINATOR ORA-16116: 使用できる作業はありません 3734535493
ANALYZER ORA-16116: 使用できる作業はありません 3734559504
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535506　★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535504　★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535499　★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535494　★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535511　★ORA-16113
READER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734560999
BUILDER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734559504
PREPARER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734559503
===========================

丁……
好多ORA-16113啊！

5.从MOS寻找关于ORA-16113的文档

[参考情報]

　ORA-16113: applying change to table or sequence "SYS"."AUD$" (Doc ID 1964125.1)

===========================
Symptoms　★事象

Archives stoped applying on logical standby server

Checking apply with this query we find it is stuck on the SYS.AUD$ table
SELECT type, status, high_scn FROM v$logstdby;

TYPE STATUS HIGH_SCN
COORDINATOR ORA-16116: no work available 4157595068
ANALYZER ORA-16116: no work available 4157607708
APPLIER ORA-16123: transaction 22 31 76123 is waiting for commit approval 4157595083
APPLIER ORA-16123: transaction 8 25 71375 is waiting for c ommit approval 4157595075
APPLIER ORA-16123: transaction 2 27 68682 is waiting for c ommit approval 4157595077
APPLIER ORA-16113: applying change to table or sequence "SYS"."AUD$"　★很像啊
＜省略＞
Cause　★原因

Although SYS is an INTERNAL SCHEMA there are some objects that are propagated to the logical standby because they support user data.
These are some: AUD$, SEQ$, and FGA$
What usually happens is the SYS.AUD$ table gets too big and then it is too slow to be updated in the logical standby.　★SYS.AUD$ table肥大化的话，就有可能发生哦
You can skip the object or reorg it in the primary.

Solution　★解決方法
＜省略＞
If not using broker
ALTER DATABASE STOP LOGICAL STANDBY APPLY;　★

SQL> exec dbms_logstdby.skip('DML','SYS','AUD$');　★
SQL> exec dbms_logstdby.skip('SCHEMA_DDL','SYS','AUD$');　★

ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;　★
===========================
 
这不就是她嘛！！！！

让客户实验了一下上面的解决方法，客户的现象就解决了。

LOGICAL STANDBY的CASE已经不多，客户更多的还是使用PHYSICAL STANDBY。
遇到LOGICAL STANDBY的时候发现，可以参照下面的MOS文档，个人感觉很有帮助。

Troubleshooting Logical Standby (Doc ID 215020.1)

[Oracle][DATAGUARD] LOGICAL STANDBY环境里，有些SEQUENCE无法应用，导致Primary和Standby无法同期的更多相关文章

  1. [Oracle][DATAGUARD] PHYSICAL STANDBY环境里，使用CATALOG管理Primary和Standby

    1.先使用控制文件构筑好PHYSICAL STANDBY环境(Primary:Single 11.2.0.4,Standby Single 11.2.0.4) 2.构筑好Catalog用的服务器(Ca ...

  2. [Oracle][DATAGUARD] PHYSICAL STANDBY环境里，11.2.0.4 , 也可以使用Pfile来运行Primary和Standby（虽然很少有人用）

    ####Primary#### [oracle@primary ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on 金 ...

  3. Oracle DataGuard 升级 [11.2.0.1 -> 11.2.0.4]

    Oracle DataGuard 升级 [11.2.0.1 -> 11.2.0.4] Primary: 11.2.0.1 单机,Site A. Standby: 11.2.0.1 单机,Site ...

  4. Oracle Dataguard之物理standby的基本配置

    尽管网上有很多Oracle Dataguard的配置教程,但不难发现,很多采用的是rman duplicate这种方法,尽管此种方法较为简便.但在某种程度上,却也误导了初学者,虽说也能配置成功,但只知 ...

  5. Oracle Dataguard Standby Redo Log的两个实验

    在Data Guard环境中,Standby Redo Log是一个比较特殊的日志类型.从最新的DG安装指导中,都推荐在Primary和Standby端,都配置Standby Redo Log. 简单 ...

  6. Oracle DataGuard 物理Standby 搭建(上)

    物理standby database 环境搭建 Arch asysnc Oracle Dataguard host IP Oracle_sid DB_unique_name FAL_server FA ...

  7. Oracle DataGuard 物理Standby 搭建(下)

    主备库切换 Switchover 一般SWITCHOVER切换都是计划中的切换,特点是在切换后,不会丢失任何的数据,而且这个过程是可逆的,整个DATA GUARD环境不会被破坏,原来DATA GUAR ...

  8. Oracle Dataguard之switchover

    Oracle Dataguard的角色转换包含两类:Switchover和Failover.Switchover指主备之间角色转换,主库降为备库,备库升级为主库.而failover则是指主库出现问题时 ...

  9. Oracle DataGuard数据备份方案详解

    Oracle DataGuard是一种数据库级别的HA方案,最主要功能是冗灾.数据保护.故障恢复等. 在生产数据库的"事务一致性"时,使用生产库的物理全备份(或物理COPY)创建备 ...

随机推荐

  1. SAP 查询分析器，查询报表自动生成，SQL查询测试实现说明（转）

    在日常的SAP开发和应用中,经常需要通过查询SAP数据表来处理日常业务,比如:数据对账.报表SQL测试.SAP查询功能开发等.通过开发SAP查询分析器,SAP实施和开发人员,可以在较短的时间内查询到需 ...

  2. Vue学习(一)Vue目录结构

    安装教程网上一大把,可以自己搜索.记录下学习过程. 认识下Vue的目录结构,取自:https://www.cnblogs.com/dragonir/p/8711761.html vue 文件目录结构详 ...

  3. TeamView工具在提示“似乎用于商用环境”的处理方式

    第一种:通过修改这个文件/也有可能删除(没有尝试过)

  4. SPOJ 1811 LCS - Longest Common Substring

    思路 和SPOJ 1812 LCS2 - Longest Common Substring II一个思路,改成两个串就有双倍经验了 代码 #include <cstdio> #includ ...

  5. yii 动态增加路由规则

    使用 UrlManager类的addRules方法. 如 $url->addRules([ 'zzz/<id:\d+>' => '/test/hi', ]); 这个规则必须在开 ...

  6. java中==和equals的区别（转）

    java中的数据类型,可分为两类: 1.基本数据类型,也称原始数据类型.byte,short,char,int,long,float,double,boolean   他们之间的比较,应用双等号(== ...

  7. winform SerialPort串口通信问题

    一.串口通信简介串行接口(串口)是一种可以将接受来自CPU的并行数据字符转换为连续的串行数据流发送出去,同时可将接受的串行数据流转换为并行的数据字符供给CPU的器件.一般完成这种功能的电路,我们称为串 ...

  8. 方差分析 | ANOVA | 原理 | R代码 | 进阶 | one way and two way

    原理 比较两组就用t-test,比较三组及以上就用ANOVA.注意:我们默认说的都是one way ANOVA,也就是对group的分类标准只有一个,比如case和control(ABCD多组),tw ...

  9. 理解不为人知的ClassLoader

    JAVA类装载方式,有两种: 1.隐式装载, 程序在运行过程中当碰到通过new 等方式生成对象时,隐式调用类装载器加载对应的类到jvm中. 2.显式装载, 通过class.forname()等方法,显 ...

  10. web前端bug积累

    在将img转换为canvas时,在firefox中,如果img引用的svg使用了fill=(#id),此fill是无效的,可能导致整个转换失效,html2canvas.js也不行.chrome有效,其 ...