kettle之mongodb数据同步

需求：

1.源数据库新增一条记录，目标库同时新增一条记录；

2.源数据库修改一条记录，目标库同时修改该条记录；

示例用到三个Kettle组件

下面详细说下每个组件的配置

Source:

本示例连接的是Mongodb数据库，四个字段，ID默认为主键，_id会系统自动生成暂时先不管。

本节点的详细说明，可见官网：http://wiki.pentaho.com/display/EAI/MongoDB+Input

值映射：

本步在本示例作用不大，只是为了测试效果。按照截图上进行配置即可

MongoDbOutput:

关键是这一步的配置

官网上对这个tab页的解释是这样的：

2.2 Selecting the write mode

The MongoDb output step provides a number of options that control what and how data is written to the target Mongo document collection. By default, data is inserted into the target collection. If the specified collection doesn't exist, it will be created before data is inserted. Selecting the Truncate option will delete any existing data in the target collection before inserting begins. Unless unique indexes are being used (see section on indexing below) then Mongo DB will allow duplicate records to be inserted. Mongo DB allows for fast bulk insert operations - the batch size can be configured using the Batch insert size field. If no value is supplied here, then the default size of 100 rows is used.

Selecting the Upsert option changes the write mode from insert to upsert (i.e. update if a match is found, otherwise insert a new record). Information on defining how records are matched can be found in the next section. Standard upsert replaces a matched record with an entire new record based on all the incoming fields specified in the Mongo document fields tab. Modifier update enables modifier ($ operators) to be used to mutate individual fields within matching documents. This type of update is fast and involves minimal network traffic; it also has the ability to update all matching documents, rather than just the first, if the Multi-update option is enabled

个人理解就是默认都不选择时往目标集合添加数据，如果指定的集合不存在则先创建集合然后插入数据；

Truncate:插入数据之前先把集合里的所有数据删除。（相当于数据删除同步）；

Update:修改数据，存在则修改，不存在不操作。

upsert:找到匹配项则修改，找不到匹配项则添加。

勾选上红色圈着的选项之后，源数据修改、添加了，在目标库里都会有相应的操作。不过还要设置下面的一步

ID为主键match field for update时一定要选择Y否则运行时出错。

同步过程最主要的就是上边列出的几步设置，当然如果想要再设置更强大的功能，可详细去研究官网的API

官网API地址：http://wiki.pentaho.com/display/EAI/

示例kri文件下载地址：http://files.cnblogs.com/nyzhai/mongodbTran.rar