Oracle GoldenGate Best Practices: Active-Active Configuration with DML Auto CDR
Executive Overview
This document is an introduction to Oracle GoldenGate (DIPC remote agent)’s best practices and guidelines for configuring OGG in an active-active environment using the built in functionality of DML Auto Conflict Detection and Resolution (Auto CDR), LOB data type column is also take into consideration.
Prerequisites
· RDBMS is Oracle Database 12.2+
· GoldenGate version 12.3/DIPC agent 18.2.3
· OS: All Oracle GoldenGate supported platforms for Oracle
Introduction
Setting up Conflict Management is key to the OGG bi-direction data sync. Most Active-Active or Multi-Master Systems will have some form of conflict. Conflicts happen due to at least one row being modified on more than one system.
Prior to RDBMS 12.2 and OGG12.2
Tables should contain timestamp columns, or we have to add column onto the table likewise to achieve the same goal. The timestamp column i.e. CHANGE_TS is used for Conflict management and decide resolution of conflict when it happen.
CDR steps are as following:
Datatypes that can be compared are supported:
• NUMERIC
• DATE
• TIMESTAMP
• CHAR/NCHAR
• VARCHAR/NVARCHAR
Typically these data types are used with parameters like COMPARECOLS, GETBEFORECOLS, and in resolution parameters using RESOLVECONFLICT ([USEMIN] | [USEMAX]).
LOGALLSUPCOLS
Extract captures before images for UPDATE operations.
Extract captures before images of supplemental logged columns for both UPDATE and DELETE operations.
GETBEFORECOLS
Ensures certain columns are logged
TABLE option in extract
Example:
GETBEFORECOLS (ON UPDATE ALL, ON DELETE ALL)
TRANSLOGOPTION EXCLUDETAG
• Tag supplied to transaction as it is extracted to prevent receiving replicat from trying to send it back to source
• Classic/Integrated Extract (primary)
• Tag can be any number/letter [0-9 A-Z]
Example:
TRANSLOGOPTION EXCLUDETAG 0294
COMPARECOLS
Used by Replicat to detect and resolve update/delete conflicts
Example:
COMPARECOLS (ON UPDATE ALL, ON DELETE ALL)
Since RDBMS 12.2 and OGG12.3
Timestamp column can be added without effecting the application or code base- leverages invisible columns/RDBMS 12.2 kernel.
Auto CDR makes the conflict management much simpler than before, you also need to meet below prerequisites
· Requires source and target to be Oracle Database 12.2
· Requires Integrated Extract (Classic Extract does not support invisible columns)
· Requires Integrated Replicat or Integrated Parallel Replicat
No Visible Application Changes:
• Easy to enable
• Invisible timestamp columns automatically created and maintained by database kernel
• Delete tombstone log to support updates-delete conflicts (new capability)
• Enables supplemental logging on required columns
• Allows ability to detect conflicts for LOB column changes
• No need to specify CDR/REPERROR parameters
• Last record wins (highest timestamp)
Enable by ADD_AUTO_CDR for LOB columns
SQL> BEGIN
dbms_goldengate_adm.add_auto_cdr(
schema_name => 'HR',
table_name => 'EMPLOYEES');
END;
/
PL/SQL procedure successfully completed.
The procedure crates an invisible timestamp column to associate with the row, (If no LOB column exist in table). If we have multiple LOB columns, i.e. 2 LOB columns, it will create 2 separate invisible timestamp columns to associate with each LOB columns and one invisible timestamp column associate with the other columns by default.
If we have a test table with below structure:
After enable the Auto CDR by:
SQL> BEGIN
DBMS_GOLDENGATE_ADM.ADD_AUTO_CDR(
schema_name => 'hr',
table_name => 'test');
END;
/
PL/SQL procedure successfully completed.
We query the ALL_GG_AUTO_CDR_COLUMNS table
SQL>SELECT TABLE_OWNER,
TABLE_NAME,
COLUMN_GROUP_NAME,
COLUMN_NAME,
RESOLUTION_COLUMN
FROM ALL_GG_AUTO_CDR_COLUMNS
ORDER BY TABLE_OWNER, TABLE_NAME;
We can see 3 CDRTS timestamp columns were created, 2 for LOB columns and one for the other columns.
By enable the Auto CDR, we
• Automatically creates an invisible timestamp column to associate with the row
• Invisible timestamp column automatically maintained
• Automatically enables Delete Tombstone Log Table
• Tombstone table automatically maintained
• Automatically enables all column supplemental logging
Monitoring
We can get the statics for CDR resolution by
GGCSI> stats replicat <group name>,reportcdr
From my test till now, with OGG Version 12.3.0.1.2 OGGCORE_12.3.0.1.0_PLATFORMS_171208.0005_FBO and Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production, LOB CDR resolution stats is NOT show here.
Steps for Implementing OGG in an active-active configuration
Enable Oracle GoldenGate on database
1. Open archive log mode
[Source] [Target]
SQL>shutdown immediate;
SQL>startup mount;
SQL>alter database archivelog;
SQL>alter database open;
SQL>alter system switch logfile;
SQL>archive log list;
SQL> alter PLUGGABLE database all open;
SQL>create user c##ogg identified by ogg;
SQL>GRANT DBA to c##ogg container=all;
SQL> exec dbms_goldengate_auth.grant_admin_privilege('c##ogg',container=>'ALL');
2. Open supplemental log data and force logging
[Source] [Target]
SQL>SELECT supplemental_log_data_min, force_logging FROM v$database;
SQL>alter database add supplemental log data;
SQL>alter database force logging;
SQL>SELECT supplemental_log_data_min, force_logging FROM v$database;
3. Enabling Oracle GoldenGate in the Database
[Source] [Target]
SQL> alter system set enable_goldengate_replication = true scope=both;
5. Create OGG Users
[Source] [Target]
SQL> create user ogg identified by ogg;
SQL>GRANT DBA to ogg;
SQL> BEGIN
dbms_goldengate_auth.grant_admin_privilege
(
grantee => 'c##ogg',
privilege_type => 'CAPTURE',
grant_select_privileges => TRUE
);
END;
/
[Source] [Target]
SQL>BEGIN
dbms_goldengate_auth.grant_admin_privilege
(
grantee => 'c##ogg',
privilege_type => 'APPLY',
grant_select_privileges => TRUE
);
END;
/
Install Oracle GoldenGate
We use the DIPC remote agent version 18.2.3 in this manual,
just download it and unzip, we can use it.
1. Update the tnsnames.ora
tnsnames.ora
SRC =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =
192.168.56.106)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = PDBSRC)
)
)
TGT =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =
192.168.56.106)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = PDBTGT)
)
)
ORCL =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =
192.168.56.106)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = ORCL)
)
)
2. Update the .bash_profile for OCI
.bash_profile
#
.bash_profile
# Get the
aliases and functions
# User
specific environment and startup programs
ORACLE_HOME=/u01/src/oci; export ORACLE_HOME
export
JAVA_HOME=/usr/java/jdk1.8.0_92
export
JRE_HOME=/usr/java/jdk1.8.0_92/jre
PATH=$JAVA_HOME/bin:$ORACLE_HOME/bin:$PATH
; export PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$JAVA_HOME/lib:
LD_LIBRARY_PATH=$ORACLE_HOME:/lib:/usr/lib; export
LD_LIBRARY_PATH
export
TNS_ADMIN='/u01/src/oci/network/admin/'
3. Testing the Connection
Setting up OGG
OGG Parameters on the Source:
1. Manager:
PORT 8809
2. Extracts:
-- Extract :extsrc
extract
extsrc
userid
c##ogg@orcl,password ogg
exttrail
./dirdat/es
TranLogOptions
Excludetag 00
table
pdbsrc.hr.*;
--Pump: pumsrc
extract
pumsrc
passthru
rmthost
<ip>,mgrport 8919
rmttrail
./dirdat/rs
table
pdbsrc.hr.*;
3. Replicat:
--Replicat: reptgt
replicat
reptgt
userid
ogg@src, password ogg
discardfile
reptgt.dsc, append
NOMAPINVISIBLECOLUMNS
map
pdbtgt.hr.test, target pdbsrc.hr. test;
map
pdbtgt.hr.jobs, target pdbsrc.hr. jobs;
OGG Parameters on the Target:
1. Manager:
PORT
8919
2. Extracts:
Extract exttgt:
extract
exttgt
userid
c##ogg@orcl,password ogg
exttrail
./dirdat/et
TranLogOptions
Excludetag 00
table
pdbtgt.hr.*;
Pump: pumtgt
extract
pumtgt
passthru
rmthost
<ip>,mgrport 8809
rmttrail
./dirdat/rt
table
pdbtgt.hr.*;
3. Replicat:
replicat
repsrc
userid
ogg@tgt, password ogg
discardfile
repsrc.dsc, append
NOMAPINVISIBLECOLUMNS
map
pdbtgt.hr.test, target pdbsrc.hr. test;
map
pdbtgt.hr.jobs, target pdbsrc.hr. jobs;
Adding groups:
Source:
GGCSI> obey Src.oby
--Source OGG
Setup
add extract
extsrc, integrated tranlog, begin now
add exttrail
./dirdat/es, extract extsrc
dblogin
userid c##ogg@orcl password ***
register
extract extsrc database container(pdbsrc)
add
schematrandata PDBSRC.HR,allcols
info
trandata PDBSRC.HR.*
add extract
pumsrc, exttrailsource ./dirdat/es
add rmttrail
./dirdat/rs, extract pumsrc
add replicat
reptgt, integrated ,exttrail ./dirdat/rt
Target:
GGCSI>
obey Tgt.oby
----Target
OGG Setup
add extract exttgt,integrated
tranlog, begin now
add exttrail
./dirdat/et, extract exttgt
dblogin
userid c##ogg@orcl password ***
register
extract exttgt database container(pdbtgt)
add
schematrandata pdbtgt.hr,allcols
info
trandata pdbtgt.hr.*
add extract
pumtgt, exttrailsource ./dirdat/et
add rmttrail
./dirdat/rt,extract pumtgt
add replicat
repsrc, integrated ,exttrail ./dirdat/rs
Start the processes
Source and Target:
GGSCI>
start mgr !
GGSCI>
start er *
Source:
Target:
Testing Steps for Auto-CDR
In all our test cases below, we took the DML against the
source database firstly and then against target in very short time.
Insert Conflict
Source:
And then on Target:
Query the source again:
Review in GGSCI:
Update Conflict
This time we are going to test a table with LOB data type
column,
INSERT INTO hr.test
VALUES(3,'Pic No.1',bfilename('IMAGE_DIR','1.png')); commit;
What we get on the source and target:
Source:
Target:
Update the LOB value in table:
Source:
update hr.test set
IMAGE =(bfilename('IMAGE_DIR','2.png')),MEMO='PIC #2' where id=3; commit;
Target:
update hr.test set
IMAGE =(bfilename('IMAGE_DIR','3.png')),MEMO='PIC #3' where id=3; commit;
We also query the sample data for those 2 LOB column
For Pic#2:
For Pic#3:
Query source and target we get:
We did not get the CDR stats for LOB columns by running:
stats
<Group_name>,reportcdr
Delete Conflict
1. Run the SQL to delete the record on both source
and target:
delete from
hr.emp where empno=1; commit;
2. Query the result on both databases:
select
EMPNO,SAL,CDRTS$ROW from hr.emp where
empno=1;
No record returns
3. Review in OGG
Source:
Target:
Conclusion
This paper covers the OGG Auto CDR feature which introduced
in version 12.3.
• It
has No Visible Application Changes and more simple to handle.
• It
can handle to detect conflicts for LOB column changes.
• A
Last record wins (with highest timestamp).
If you want to customize the conflict detection resolution
more complex, not only highest timestamp win, you can still try classic CDR. Please
refer to the OGG Administrative Guide for a more extensive view of CDR.
Reference
1. Oracle GoldenGate Best Practices: Active-Active
Configuration with DDL and DML CDR --Tracy West
2. Oracle GoldenGate 12.3: Conflict, Detection and
Resolution –Bobby Curtis
https://www.slideshare.net/BobbyCurtisMBA/goldengate-cdr-from-ukoug-2017
3. CDR Info:
https://docs.oracle.com/goldengate/c1230/gg-winux/GGODB/automatic-conflict-detection-and-resolution2.htm#GGODB-GUID-29EDF7BF-6BD8-4E02-98AB-DDBB4719D038
Oracle GoldenGate Best Practices: Active-Active Configuration with DML Auto CDR的更多相关文章
- Oracle Grid control 11g及Active DataGuard 11g安装部署
Oracle Grid control 11g及Active DataGuard 11g安装部署(一) 原贴 http://blog.csdn.net/lichangzai/article/detai ...
- Oracle GoldenGate for Oracle 11g to PostgreSQL 9.2.4 Configuration
Installing and setting up Oracle GoldenGate connecting to an Oracle database Also please make sure t ...
- zabbix 启到不起来:active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)
cat /var/log/zabbix_agent_log 查看日记出现报错:active check configuration update from [127.0.0.1:10051] star ...
- Oracle GoldenGate实现数据库同步
前言:最近刚好在弄数据库同步,网上查了些资料再加上自己整理了一些,做个分享! 一.GoldenGate的安装 1.安装包准备 数据库版本:Oracle Database 11g Release 2(1 ...
- oracle goldengate 远程捕获和投递
很早之前,OGG只支持部署在数据库主机上,这叫本地化部署.而现在OGG支持远端部署,即OGG软件不安装在数据库主机上,而是安装在单独的机器上,负责数据抽取和投递. 这样做的好处: l 易于管理 - 在 ...
- Oracle GoldenGate 二、配置和使用
Oracle GoldenGate 二.配置和使用 配置和使用GoldenGate的步骤 1 在源端和目标端配置数据库支持GoldenGate 2 在源端和目标端创建和配置GoldenGate实例 3 ...
- Oracle GoldenGate
Oracle GoldenGate实现数据库同步 前言:最近刚好在弄数据库同步,网上查了些资料再加上自己整理了一些,做个分享! 一.GoldenGate的安装 官方文档: Oracle®GoldenG ...
- oracle ogg 单实例双向复制搭建(oracle-oracle)--Oracle GoldenGate
oracle ogg 单实例双向复制搭建(oracle-oracle)--Oracle GoldenGate --继昨天的测试,这一篇实施单实例双向复制(完全重新搭建) --环境不变 db1,db2( ...
- Streaming data from Oracle using Oracle GoldenGate and Kafka Connect
This is a guest blog from Robin Moffatt. Robin Moffatt is Head of R&D (Europe) at Rittman Mead, ...
随机推荐
- 备战省赛组队训练赛第十四场(UPC)
codeforces:传送门 upc:传送门 外来题解: [1]:https://blog.csdn.net/ccsu_cat/article/details/86707446 [2]:https:/ ...
- Visio文本相关操作
三种方式:双击形状输入文本,插入文本框, 文本工具 文本块工具 选择后可以对文本进行移动旋转 如果要给文本加入边框 直接显示线条就可以了 因为都是文本框 添加特殊文本: 插入符号 插入域 比如当前时间 ...
- CodeTypeDeclaration,CodeMemberProperty动态生成代码
由于是CodeDom些列,所以先介绍几个CodeDom表达式: :CodeConditionStatement:判断语句即是if(condition){} else{},看最全的那个构造函数: pub ...
- 基于 HTML5 + WebGL 的无人机 3D 可视化系统
前言 近年来,无人机的发展越发迅速,既可民用于航拍,又可军用于侦察,涉及行业广泛,也被称为“会飞的照相机”.但作为军事使用,无人机的各项性能要求更加严格.重要.本系统则是通过 Hightopo 的 ...
- 本地安装配置Gradle及IDEA使用本地Gradle
一.下载Gradle 下载地址:http://services.gradle.org/distributions/ 下载版本的bin.zip 二.配置环境变量 三.验证 在cmd模式下查看 ...
- MySQL基础之事务编程学习笔记
MySQL基础之事务编程学习笔记 在学习<MySQL技术内幕:SQL编程>一书,并做了笔记.本博客内容是自己学了<MySQL技术内幕:SQL编程>事务编程一章之后,根据自己的理 ...
- 只用这 6 个字符,就可以写出任意 JavaScript 代码!
你可能在网上见过有人用 几个不同的字符写的各种稀奇古怪的 JavaScript 代码,虽然看起来奇怪,但是能正常运行!比如这个: (!(~+[])+{})[--[~+""][+[] ...
- C# 对象与引用变量
从宏观的角度来看,对象是类的实例.比如: //定义一个名为Someone的类,代表这么一些人(通过指定年龄,性别,性格等基本信息)class Someone { public int age; p ...
- wide&deep模型演化
推荐系统模型演化 LR-->GBDT+LR FM-->FFM-->GBDT+FM|FFM FTRL-->GBDT+FTRL Wide&DeepModel (Deep l ...
- win服务器管理工具,服务器vps管理
win系列服务器,vps桌面如何管理?用这个工具: IIS7远程桌面批量管理,同时管理上千台vps,服务器,3389远程端口.