debezium mongodb 集成测试

debezium 是一个方便的cdc connector 可以帮助我们解决好多数据实时变更处理、数据分析、微服务的数据通信
从上次跑简单demo到现在，这个工具是有好多的变更，添加了好多方便的功能，支持了越来越多的数据库。
demo 使用了官方提供的docker-compose 文件

环境准备

• docker-compose 文件

 

version: '2'

services:

  zookeeper:

    image: debezium/zookeeper:0.9

    ports:

     - 2181:2181

     - 2888:2888

     - 3888:3888

  kafka:

    image: debezium/kafka:0.9

    ports:

     - 9092:9092

    links:

     - zookeeper

    environment:

     - ZOOKEEPER_CONNECT=zookeeper:2181

  mongodb:

    image: debezium/example-mongodb:0.9

    hostname: mongodb

    ports:

     - 27017:27017

    environment:

     - MONGODB_USER=debezium

     - MONGODB_PASSWORD=dbz

  connect:

    image: debezium/connect:0.9

    ports:

     - 8083:8083

    links:

     - kafka

     - mongodb

    environment:

     - BOOTSTRAP_SERVERS=kafka:9092

     - GROUP_ID=1

     - CONFIG_STORAGE_TOPIC=my_connect_configs

     - OFFSET_STORAGE_TOPIC=my_connect_offsets

• 启动

docker-compose up -d

 

• 加载mongodb 数据
  demo 镜像内置了一个mongo 数据的脚本

 

docker-compose -f docker-compose-mongodb.yaml exec mongodb bash -c '/usr/local/bin/init-inventory.sh'

 

 

连接的用户账户密码： debezium dbz
效果

• 启动connector

 

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-mongodb.json

 

register-mongodb.json 内容

{

    "name": "inventory-connector",

    "config": {

        "connector.class" : "io.debezium.connector.mongodb.MongoDbConnector",

        "tasks.max" : "1",

        "mongodb.hosts" : "rs0/mongodb:27017",

        "mongodb.name" : "dbserver1",

        "mongodb.user" : "debezium",

        "mongodb.password" : "dbz",

        "database.whitelist" : "inventory",

        "database.history.kafka.bootstrap.servers" : "kafka:9092"

    }

}

 

效果

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connect

ors/ -d @register-mongodb.json

​

HTTP/1.1 201 Created

Date: Tue, 25 Dec 2018 12:15:59 GMT

Location: http://localhost:8083/connectors/inventory-connector

Content-Type: application/json

Content-Length: 379

Server: Jetty(9.4.12.v20180830)

​

{"name":"inventory-connector","config":{"connector.class":"io.debezium.connector.mongodb.MongoDbConnector","tasks.max":"1","mongodb.hosts":"rs0/mongodb:27017","mongodb.name":"dbserver1","mongodb.user":"debezium","mongodb.password":"dbz","database.whitelist":"inventory","database.history.kafka.bootstrap.servers":"kafka:9092","name":"inventory-connector"},"tasks":[],"type":null}%

 

 

测试

• 打开消费者（kafka 命令行）

docker-compose -f docker-compose-mongodb.yaml exec kafka /kafka/bin/kafka-console-consumer.sh \

    --bootstrap-server kafka:9092 \

    --from-beginning \

    --property print.key=true \

    --topic dbserver1.inventory.customers

 

• 修改数据&&查看效果
  修改数据：
  
  效果

 

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.customers.Key"},"payload":{"id":"1001"}} {"schema":{"type":"struct","fields":[{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"after"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"patch"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":false,"field":"rs"},{"type":"string","optional":false,"field":"ns"},{"type":"int32","optional":false,"field":"sec"},{"type":"int32","optional":false,"field":"ord"},{"type":"int64","optional":true,"field":"h"},{"type":"boolean","optional":true,"default":false,"field":"initsync"}],"optional":false,"name":"io.debezium.connector.mongo.Source","version":1,"field":"source"},{"type":"string","optional":true,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.customers.Envelope"},"payload":{"after":null,"patch":"{\"$v\" : 1,\"$set\" : {\"first_name\" : \"dalong\"}}","source":{"version":"0.9.0.Beta2","connector":"mongodb","name":"dbserver1","rs":"rs0","ns":"inventory.customers","sec":1545740280,"ord":1,"h":5866229044893041819,"initsync":false},"op":"u","ts_ms":1545740280088}}

 

说明

debezium mongodb 的cdc 是基于复制集实现的，实际上mongodb 已经支持了stream 可以进行数据的捕获处理

参考资料

https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mongodb
https://debezium.io/