yugabyte cdc 试用

目前yugabyte 的cdc 功能处于beta 阶段，应该也快实际可用了，以下是一个简单的试用

环境准备

使用docker-compose

• docker-compose 文件

version: '2'

services:

  cdc-connector:

      build: ./

  graphql-engine:

    image: hasura/graphql-engine:v1.0.0-beta.6

    ports:

    - "8080:8080"

    environment:

      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres@yb-tserver2:5433/postgres

      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console

      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log

  yb-master:

      image: yugabytedb/yugabyte:latest

      container_name: yb-master-n1

      command: [ "/home/yugabyte/bin/yb-master", 

                "--fs_data_dirs=/mnt/disk0,/mnt/disk1", 

                "--master_addresses=yb-master-n1:7100", 

                "--replication_factor=1"]

      ports:

      - "7000:7000"

      environment:

        SERVICE_7000_NAME: yb-master

  yb-tserver:

      image: yugabytedb/yugabyte:latest

      container_name: yb-tserver-n1

      command: [ "/home/yugabyte/bin/yb-tserver", 

                "--fs_data_dirs=/mnt/disk0,/mnt/disk1",

                "--start_pgsql_proxy", 

                "--tserver_master_addrs=yb-master-n1:7100"]

      ports:

      - "9042:9042"

      - "6379:6379"

      - "5433:5433"

      - "9000:9000"

      environment:

        SERVICE_5433_NAME: ysql

        SERVICE_9042_NAME: ycql

        SERVICE_6379_NAME: yedis

        SERVICE_9000_NAME: yb-tserver

      depends_on:

      - yb-master

  yb-tserver2:

      image: yugabytedb/yugabyte:latest

      container_name: yb-tserver-n2

      command: [ "/home/yugabyte/bin/yb-tserver", 

                "--fs_data_dirs=/mnt/disk0,/mnt/disk1",

                "--start_pgsql_proxy", 

                "--tserver_master_addrs=yb-master-n1:7100"]

      ports:

      - "9043:9042"

      - "6380:6379"

      - "5434:5433"

      - "9001:9000"

      environment:

        SERVICE_5433_NAME: ysql

        SERVICE_9042_NAME: ycql

        SERVICE_6379_NAME: yedis

        SERVICE_9000_NAME: yb-tserver

      depends_on:

      - yb-master

​

 

 

• 启动ysql

docker-compose exec yb-master bash -c "YB_ENABLED_IN_POSTGRES=1 FLAGS_pggate_master_addresses=yb-master-n1:7100 /home/yugabyte/postgres/bin/initdb -D /tmp/yb_pg_initdb_tmp_data_dir -U postgres"

• 创建数据库

CREATE TABLE users (name text, pass text, id int, primary key (id));

 

• 下载connector

https://github.com/yugabyte/yb-kafka-connector/blob/master/yb-cdc/yb-cdc-connector.jar

• 安装connector 到java ext 目录(本地)
  以上demo使用的是docker 运行

 

Linux: jre\lib\ext\yb-cdc-connector.jar

macOS: \Library\Java\Extensions\yb-cdc-connector.jar

Windows: %SystemRoot%\Sun\Java\lib\ext\yb-cdc-connector.jar

简单测试

没有使用kafka，使用的是 标准输出，为了简化测试 ，实际生产肯定应该使用kafaka

• 启动命令 （本地）
  ```code
  java -jar yb_cdc_connector.jar --table_name postgres --master_addrs 127.0.0.1:7100 --log_only

 

* 启动模式

上边docker-compose up  -d 之后会包含yb-cdc-connector

效果

```code

/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.3) or chardet (2.2.1) doesn't match a supported version!

  RequestsDependencyWarning)

Attaching to app_cdc-connector_1

cdc-connector_1 | [2019-10-18 02:36:00,907] INFO Starting CDC Kafka Connector... (org.yb.cdc.Main:28)

cdc-connector_1 | 2019-10-18 02:36:00,913 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Creating new YB client...

cdc-connector_1 | [2019-10-18 02:36:01,028] WARN Slow DNS lookup! Resolved IP of `yb-master' to 172.19.0.4 in 6093014ns (org.yb.client.AsyncYBClient:2102)

cdc-connector_1 | [2019-10-18 02:36:01,171] INFO Discovered tablet YB Master for table YB Master with partition ["", "") (org.yb.client.AsyncYBClient:1593)

cdc-connector_1 | 2019-10-18 02:36:02,195 [ERROR|org.yb.cdc.KafkaConnector|KafkaConnector] Could not find a table with name users

cdc-connector_1 | [2019-10-18 02:37:13,641] INFO Starting CDC Kafka Connector... (org.yb.cdc.Main:28)

cdc-connector_1 | 2019-10-18 02:37:13,648 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Creating new YB client...

cdc-connector_1 | [2019-10-18 02:37:13,742] WARN Slow DNS lookup! Resolved IP of `yb-master' to 172.19.0.4 in 3880047ns (org.yb.client.AsyncYBClient:2102)

cdc-connector_1 | [2019-10-18 02:37:13,856] INFO Discovered tablet YB Master for table YB Master with partition ["", "") (org.yb.client.AsyncYBClient:1593)

cdc-connector_1 | 2019-10-18 02:37:13,868 [ERROR|org.yb.cdc.KafkaConnector|KafkaConnector] Could not find a table with name users

cdc-connector_1 | [2019-10-18 02:39:29,903] INFO Starting CDC Kafka Connector... (org.yb.cdc.Main:28)

cdc-connector_1 | 2019-10-18 02:39:29,909 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Creating new YB client...

cdc-connector_1 | [2019-10-18 02:39:30,003] WARN Slow DNS lookup! Resolved IP of `yb-master' to 172.19.0.4 in 3878725ns (org.yb.client.AsyncYBClient:2102)

cdc-connector_1 | [2019-10-18 02:39:30,125] INFO Discovered tablet YB Master for table YB Master with partition ["", "") (org.yb.client.AsyncYBClient:1593)

cdc-connector_1 | 2019-10-18 02:39:30,137 [ERROR|org.yb.cdc.KafkaConnector|KafkaConnector] Could not find a table with name users

cdc-connector_1 | [2019-10-18 02:41:15,985] INFO Starting CDC Kafka Connector... (org.yb.cdc.Main:28)

cdc-connector_1 | 2019-10-18 02:41:15,991 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Creating new YB client...

cdc-connector_1 | [2019-10-18 02:41:16,089] WARN Slow DNS lookup! Resolved IP of `yb-master' to 172.19.0.4 in 3907821ns (org.yb.client.AsyncYBClient:2102)

cdc-connector_1 | [2019-10-18 02:41:16,208] INFO Discovered tablet YB Master for table YB Master with partition ["", "") (org.yb.client.AsyncYBClient:1593)

cdc-connector_1 | 2019-10-18 02:41:16,285 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Created new stream with id b0d1f019a5544fee8e4f0171f64e453b

cdc-connector_1 | 2019-10-18 02:41:16,291 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Polling for new tablet bf9fff471e8044cdaef702f0e7489741

cdc-connector_1 | 2019-10-18 02:41:16,292 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Polling for new tablet 66991b7373294418977409d50499ea5d

cdc-connector_1 | 2019-10-18 02:41:16,292 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Polling for new tablet 

cdc-conne

 

• 添加数据

INSERT INTO "public"."users"("name", "pass", "id") VALUES('dalong', 'eee', 1);

• 效果

说明

yugabyte cdc 对于实时数据处理，数据分析，数据传递，还是很方便的，基于此功能我们可以开发灵活的数据分析系统 ，当前还是beta 状态，很期待
可以ga

参考资料

https://docs.yugabyte.com/latest/deploy/cdc/cdc-to-kafka/