PostgreSQL的OGG -- bucardo

bucardo是PostgreSQL数据库中实现双向同步的软件，可以实现PostgreSQL数据库的双master的方案，不过bucardo中的同步都是异步的，它是通过触发器记录变化，程序是perl写的.bucardo可以实现postgresql的多主复制、主从同步，甚至可以以postgresql为源库，可以和oracle、mysql、mongodb等很多数据库进行数据异步同步。

 

而pg原生的stream replication；虽可以同步；只能单向同步。只能是只读操作。而bucardo不支持DDL的同步； 所以说 相当于对stream replication的关系；相当于同oracle中 ogg对dg。

 

一 , 安装软件包

 

1. 首先安装依赖包：

   yum install perl-boolean

   wget http://bucardo.org/downloads/dbix_safe.tar.gz  

   wget http://www.cpan.org/authors/id/T/TU/TURNSTEP/DBD-Pg-3.5.0.tar.gz   

   wget http://search.cpan.org/CPAN/authors/id/T/TI/TIMB/DBI-1.630.tar.gz

 

2. 安装Bucardo

   wget http://bucardo.org/downloads/Bucardo-5.4.1.tar.gz

    对上面perl安装包；安装过程

perl Makefile.PL
make
make install

3. 上面安装好了软件；下面实例化Bucardo;在postgres用户下面新建.bucardorc 文件

log_conflict_file         = /home/postgres/log/bucardo_conflict.log
piddir                    = /home/postgres/run
reason_file               = /home/postgres/log/bucardo.restart.reason.log
warning_file              = /home/postgres/log/bucardo.warning.log

4. 初始化bucardo

[postgres@sdserver40_222 ~]$ bucardo install
This will install the bucardo database into an existing Postgres cluster.
Postgres must have been compiled with Perl support,
and you must connect as a superuser

Current connection settings:
. Host:           <none>
. Port:
. User:           postgres
. Database:       bucardo
. PID directory:  /home/postgres/run
Enter a number to change it, P to proceed, or Q to quit: 

Change the host to: 127.0.0.1

Changed host to: 127.0.0.1
Current connection settings:
. Host:           127.0.0.1
. Port:
. User:           postgres
. Database:       bucardo
. PID directory:  /home/postgres/run
Enter a number to change it, P to proceed, or Q to quit: 

Change the database name to: mydb

Changed database name to: mydb
Current connection settings:
. Host:           127.0.0.1
. Port:
. User:           postgres
. Database:       mydb
. PID directory:  /home/postgres/run
Enter a number to change it, P to proceed, or Q to quit: P

Postgres version is: 9.5
Attempting to create and populate the bucardo database and schema
Database creation is complete

Updated configuration setting "piddir"
Installation is now complete.
If you see errors or need help, please email bucardo-general@bucardo.org

You may want to check over the configuration variables next, by running:
bucardo show all
Change any setting by using: bucardo set foo=bar 

这步可能会出现问题；大多数是依赖包没装的问题；现后面FAQ

 

二， 配置bucardo环境

 

下面的配置的环境:

主机1： 192.168.1.201 

主机2： 192.168.1.202

 

1 添加源数据库

在192.168.1.201 执行

[postgres@sdserver40_222 ~]$ bucardo add db db1 dbname=mydb host=127.0.0.1 port=  user=postgres
Added database "db1"
[postgres@sdserver40_222 ~]$ bucardo add db db2 dbname=mydb host=192.168.1.202 port= user=postgres
Added database "db2"

这里需要注意的是；psql -h 192.168.1.202是否可以连通；看是不是防火墙；在主机192.168.1.202上pg_hba.conf是否配置。

 

2. 添加表：

[postgres@sdserver40_222 ~]$  bucardo add all tables

New tables added: 0

Already added: 13

[postgres@sdserver40_222 ~]$ bucardo add all sequences

Sorry, no sequences were found

New sequences added: 0

如果要添加某一个表，则可以bucardo add  table tablename

 

3.  添加复制群组（即复制一组表）：

[postgres@sdserver40_222 ~]$ bucardo add relgroup relgroup1 lottu03 lottu01;

Created relgroup "relgroup1"

The following tables or sequences are now part of the relgroup "relgroup1":

  public.lottu01

  public.lottu03

[postgres@sdserver40_222 ~]$ bucardo add dbgroup dbgroup1 db1:source db2:target

Created dbgroup "dbgroup1"

Added database "db1" to dbgroup "dbgroup1" as source

Added database "db2" to dbgroup "dbgroup1" as target

1）若添加若有表则用 all tables；

2) 只能同步存在主键的table；像这里的lottu01我是没建主键的；下面添加同步步骤会出现错误。

4. 添加同步

[postgres@sdserver40_222 ~]$ bucardo add sync sync1 relgroup=relgroup1 dbs=dbgroup1
Added sync "sync1"

   1）这里注意的是；同步到表需要主键约束；不然同步是添加不了的。

会报如下错误；这个是跟触发器有关系；bucardo是建立在触发器之上的。

Failed to add sync: DBD::Pg::st execute failed: ERROR:  Table "public.lottu01" must specify a primary key! at line . at line .
CONTEXT:  PL/Perl function "validate_sync" at /usr/local/bin/bucardo line 

  2）删除同步 :bucardo add sync syncname

  3 ) bucardo 添加同步；参数详情 如下；

  add sync:
      bucardo add sync name relgroup=xxx dbs=xxx

    Adds a sync, which is a named replication event containing information
    about what to replicate from where to where. The supported parameters
    are:

    "dbs"
        The name of a dbgroup or comma-delimited list of databases. All of
        the specified databases will be synchronized. Required.

    "dbgroup"
        The name of a dbgroup. All of the databases within this group will
        be part of the sync. If the dbgroup does not exists and a separate
        list of databases is given, the group will be created and populated.

    "relgroup"
        The name of a relgroup to synchronize. All of the tables and/or
        sequences in the relgroup will be synchronized. Required unless
        "tables" is specified.

    "tables"
        List of tables to add to the sync. This implicitly creates a
        relgroup with the same name as the sync. Required unless "relgroup"
        is specified.

    "status"
        Indicates whether or not the sync is active. Must be either "active"
        or "inactive". Defaults to "active".

    "rebuild_index"
        Boolean indicating whether or not to rebuild indexes after every
        sync. Defaults to off.

    "lifetime"
        Number of seconds a KID can live before being reaped. No limit by
        default.

    "maxkicks"
        Number of times a KID may be kicked before being reaped. No limit by
        default.

    "conflict_strategy"
        The conflict resolution strategy to use in the sync. Supported
        values:

        "bucardo_source"
            The rows on the "source" database always "win". In other words,
            in a conflict, Bucardo copies rows from source to target.

        "bucardo_target"
            The rows on the "target" database always win.

        "bucardo_skip"
            Any conflicting rows are simply not replicated. Not recommended
            for most cases.

        "bucardo_random"
            Each database has an equal chance of winning each time. This is
            the default.

        "bucardo_latest"
            The row that was most recently changed wins.

        "bucardo_abort"
            The sync is aborted on a conflict.

    "onetimecopy"
        Determines whether or not a sync should switch to a full copy mode
        for a single run. Supported values are:

        : off
        : always full copy
        : only copy tables that are empty on the target

    "stayalive"
        Boolean indicating whether or not the sync processes (CTL) should be
        persistent. Defaults to false.

    "kidsalive"
        Boolean indicating whether or not the sync child processes (KID)
        should be persistent. Defaults to false.

    "autokick"
        Boolean indicating whether or not tables in the sync should
        automatically send kick messages when they're modified. May be
        overridden by the "autokick" parameter of individual tables.

    "checktime"
        An interval specifying the maximum time a sync should go before
        being kicked. Useful for busy systems where you don't want the
        overhead of notify triggers.

    "priority"
        An integer indicating the priority of the sync. Lower numbers are
        higher priority. Currently used only for display purposes.

    "analyze_after_copy"
        Boolean indicating whether or not to analyze tables after every
        sync. Off by default. Optional.

    "overdue"
        An interval specifying the amount of time after which the sync has
        not run that it should be considered overdue. "check_bucardo_sync"
        issues a warning when a sync has not been run in this amount of
        time.

    "expired"
        An interval specifying the amount of time after which the sync has
        not run that it should be considered expired. "check_bucardo_sync"
        issues a critical message when a sync has not been run in this
        amount of time.

    "track_rates"
        Boolean indicating whether or not to track synchronization rates.

    "rebuild_index"
        Boolean indicating whether or not to rebuild indexes after every
        sync. Off by default. Optional.

    "strict_check"
        Boolean indicating whether or not to be strict when comparing tables
        in the sync. If the columns have different names or data types, the
        validation will fail. But perhaps the columns are allowed to have
        different names or data types. If so, disable "strict_check" and
        column differences will result in warnings rather than failing the
        validation. Defaults to true.

5. 启动bucardo

bucardo start
bucardo stop
bucardo restart

 

三，验证操作

  在原库操作

[postgres@sdserver40_222 ~]$ psql mydb
psql (9.5.)
Type "help" for help.
mydb=# delete from lottu03;
DELETE 

mydb=# insert into lottu03 values (1001,'bucardo');
INSERT  

在目标库查看

[postgres@sdserver40_210 ~]$ psql mydb
psql (9.5.)
Type "help" for help.

mydb=# select * from lottu03;
  id  |  name
------+---------
  | bucardo
( row)

 

四， FAQ

 Q1. 在执行bucardo install 操作出现

 Creating superuser 'bucardo'
Attempting to create and populate the bucardo database and schema

INSTALLATION FAILED! (psql:/usr/local/share/bucardo/bucardo.schema:: ERROR:  language "plperlu" does not exist
HINT:  Use CREATE LANGUAGE to load the language into the database.)  

解决方法；看提示知道需要“create language plperlu;”但是在我的环境之下

mydb=# create language plperlu;
ERROR:  could not access file "$libdir/plperl": No such file or directory

 出现这种问题不奇怪；因在我的博客http://www.cnblogs.com/lottu/p/5149191.html  安装postgres中初始化时未加--with-perl;所以安装后lib目录下缺少plperl.so文件； 

解决方法：再来一边初始化postgrs； 把plperl.so复制到对应的lib目录下即可。

postgres=# CREATE LANGUAGE plperlu;
CREATE LANGUAG

 

Q2： 解决Q1；再一次执行bucardo install 操作出现

install_driver(Pg) failed: DBI version 1.614 required--this is only version 1.609 at /usr/share/perl5/Exporter/Heavy.pm line , <> line .
Compilation failed in require at (eval ) line , <> line .

 解决方法： 这个是我开始安装依赖包时；偷懒未安装DBI 包； 提示是需要更新 DBI 包；目前最新的是1.633版本；这里我们安装16.30就行 

 下载并安装： wget http://search.cpan.org/CPAN/authors/id/T/TI/TIMB/DBI-1.630.tar.gz

 

Q3： 安装bucardo install 之后错误提示

perl: symbol lookup error: /usr/local/lib64/perl5/auto/DBD/Pg/Pg.so: undefined symbol: DBIc_AIADESTROY

解决方法： 这个缺少symbol: DBIc_AIADESTROY； 这个是Q2问题遗留下来的； 之前安装DBD_pg时是之前的DBI；现DBI更新了；需要重新安装DBD_pg

 

Q4：在启动bucardo start出现错误提示

Checking for existing processes
Can't locate boolean.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/share/perl5/Bucardo.pm line 34.
BEGIN failed--compilation aborted at /usr/local/share/perl5/Bucardo.pm line 34.
Compilation failed in require at /usr/local/bin/bucardo line 833.

解决方法： yum install perl-boolean