使用tap-minio-csv 我们可以将s3 中csv 的文件，通过singer 的target 写到不用的系统中，可以兼容
s3 的存储类型,以下是一个集成minio 的测试，将minio 中的csv 数据导入到pg中

环境准备

  • docker-compose 文件
version: "3"
services:
  s3: 
    image: minio/minio
    command: server /data
    ports: 
    - "9000:9000"
    environment:
    - "MINIO_ACCESS_KEY=dalongapp"
    - "MINIO_SECRET_KEY=dalongapp"
    volumes: 
    - "./data:/data" 
  target:
    image: postgres:9.6.11
    ports:
    - "5432:5432"
    environment:
    - "POSTGRES_PASSWORD:dalong"
 
 
  • 创建bucket 并上传文件

  • 文件格式
    my_table.csv
id,username,userage2,classinfo
1,"dalong",11,"v1"
2,"rong",29,"v2"
3,"appdemo",30,"v3"
4,"tetst",30,"v4"

my_table2.csv

id2,username2,userage3,classinfo2
7,"dalong",11,"v1"
8,"rong",29,"v2"
9,"appdemo",30,"v3"
10,"tetst",30,"v4"

tap 环境准备

  • venv 初始化
mkdir  s3-tap
python3 -m venv venv
  • 激活虚拟环境
source  venv/bin/activate
  • 安装tap
pip install tap-minio-csv
 
{
    "start_date": "2017-11-02T00:00:00Z",
    "bucket": "demo",
    "aws_access_key_id":"dalongapp",
    "aws_secret_access_key":"dalongapp",
    "endpoint_url":"http://localhost:9000",
    "tables": "[{\"search_prefix\":\"exports\",\"search_pattern\":\"my_table.csv\",\"table_name\":\"my_table\",\"key_properties\":\"id\",\"delimiter\":\",\"},{\"search_prefix\":\"exports\",\"search_pattern\":\"my_table2.csv\",\"table_name\":\"my_table2\",\"key_properties\":\"id2\",\"delimiter\":\",\"}]"
}
  • 简单说明
    我们通过 tables 可以定义文件查找的方法以及csv 的处理规则

target 配置

  • venv 初始化
mkdir  pg-target
python3 -m venv venv
  • 激活虚拟环境
source  venv/bin/activate 
  • 安装tap
pip install  target_postgres
  • 配置target
{
    "host": "localhost",
    "port": 5432,
    "dbname": "postgres",
    "user": "postgres",
    "password": "dalong",
    "schema": "copy"
}

集成使用

  • 模式发现
s3-tap/venv/bin/tap-minio-csv -c s3-tap/tap-config.json -d > catalog.json

效果

WARNING I have direct access to the bucket without assuming the configured role.
INFO Starting discover
INFO Sampling records to determine table schema.
INFO Sampling files (max files: 5)
INFO Checking bucket "demo" for keys matching "my_table.csv"
INFO exports/my_table.csv
INFO 2019-08-22 05:27:52+00:00
INFO Will download key "exports/my_table.csv" as it was last modified 2019-08-22 05:27:52+00:00
INFO Sampling exports/my_table.csv (max records: 1000, sample rate: 5)
INFO Sampled 1 rows from exports/my_table.csv
INFO exports/my_table2.csv
INFO 2019-08-22 09:01:45+00:00
INFO Found 2 files.
INFO Sampling records to determine table schema.
INFO Sampling files (max files: 5)
INFO Checking bucket "demo" for keys matching "my_table2.csv"
INFO exports/my_table.csv
INFO 2019-08-22 05:27:52+00:00
INFO exports/my_table2.csv
INFO 2019-08-22 09:01:45+00:00
INFO Will download key "exports/my_table2.csv" as it was last modified 2019-08-22 09:01:45+00:00
INFO Sampling exports/my_table2.csv (max records: 1000, sample rate: 5)
INFO Sampled 1 rows from exports/my_table2.csv
INFO Found 2 files.
INFO Finished discover
  • 启用同步配置
    主要是启用selected，在每个stream 的metadata 中
 
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": [
              "id"
            ],
+ "selected":true
          }
        }

完整如下：

{
  "streams": [
    {
      "stream": "my_table",
      "tap_stream_id": "my_table",
      "schema": {
        "type": "object",
        "properties": {
          "id": {
            "type": [
              "null",
              "integer",
              "string"
            ]
          },
          "username": {
            "type": [
              "null",
              "string"
            ]
          },
          "userage2": {
            "type": [
              "null",
              "integer",
              "string"
            ]
          },
          "classinfo": {
            "type": [
              "null",
              "string"
            ]
          },
          "_sdc_source_bucket": {
            "type": "string"
          },
          "_sdc_source_file": {
            "type": "string"
          },
          "_sdc_source_lineno": {
            "type": "integer"
          },
          "_sdc_extra": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      },
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": [
              "id"
            ],
            "selected":true
          }
        },
        {
          "breadcrumb": [
            "properties",
            "id"
          ],
          "metadata": {
            "inclusion": "automatic"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "username"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "userage2"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "classinfo"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_source_bucket"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_source_file"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_source_lineno"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_extra"
          ],
          "metadata": {
            "inclusion": "available"
          }
        }
      ]
    },
    {
      "stream": "my_table2",
      "tap_stream_id": "my_table2",
      "schema": {
        "type": "object",
        "properties": {
          "id2": {
            "type": [
              "null",
              "integer",
              "string"
            ]
          },
          "username2": {
            "type": [
              "null",
              "string"
            ]
          },
          "userage3": {
            "type": [
              "null",
              "integer",
              "string"
            ]
          },
          "classinfo2": {
            "type": [
              "null",
              "string"
            ]
          },
          "_sdc_source_bucket": {
            "type": "string"
          },
          "_sdc_source_file": {
            "type": "string"
          },
          "_sdc_source_lineno": {
            "type": "integer"
          },
          "_sdc_extra": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      },
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": [
              "id2"
            ],
            "selected":true
          }
        },
        {
          "breadcrumb": [
            "properties",
            "id2"
          ],
          "metadata": {
            "inclusion": "automatic"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "username2"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "userage3"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "classinfo2"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_source_bucket"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_source_file"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_source_lineno"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "_sdc_extra"
          ],
          "metadata": {
            "inclusion": "available"
          }
        }
      ]
    }
  ]
}
 
  • 执行同步
 s3-tap/venv/bin/tap-minio-csv -c s3-tap/tap-config.json -p catalog.json | pg-target/venv/bin/target-postgres -c pg-target/target.json
 

效果：

WARNING I have direct access to the bucket without assuming the configured role.
INFO Starting sync.
INFO my_table: Starting sync
INFO Syncing table "my_table".
INFO Getting files modified since 2017-11-02 00:00:00+00:00.
INFO Checking bucket "demo" for keys matching "my_table.csv"
INFO Table 'my_table' does not exist. Creating... CREATE TABLE copy.my_table ("_sdc_extra" jsonb, "_sdc_source_bucket" character varying, "_sdc_source_file" character varying, "_sdc_source_lineno" bigint, "classinfo" character varying, "id" character varying, "userage2" character varying, "username" character varying, PRIMARY KEY ("id"))
INFO exports/my_table.csv
INFO 2019-08-22 05:27:52+00:00
INFO Will download key "exports/my_table.csv" as it was last modified 2019-08-22 05:27:52+00:00
INFO exports/my_table2.csv
INFO 2019-08-22 09:01:45+00:00
INFO Found 2 files.
INFO Syncing file "exports/my_table.csv".
INFO Wrote 4 records for table "my_table".
INFO my_table: Completed sync (4 rows)
INFO my_table2: Starting sync
INFO Syncing table "my_table2".
INFO Getting files modified since 2017-11-02 00:00:00+00:00.
INFO Checking bucket "demo" for keys matching "my_table2.csv"
INFO Table 'my_table2' does not exist. Creating... CREATE TABLE copy.my_table2 ("_sdc_extra" jsonb, "_sdc_source_bucket" character varying, "_sdc_source_file" character varying, "_sdc_source_lineno" bigint, "classinfo2" character varying, "id2" character varying, "userage3" character varying, "username2" character varying, PRIMARY KEY ("id2"))
INFO exports/my_table.csv
INFO 2019-08-22 05:27:52+00:00
INFO exports/my_table2.csv
INFO 2019-08-22 09:01:45+00:00
INFO Will download key "exports/my_table2.csv" as it was last modified 2019-08-22 09:01:45+00:00
INFO Found 2 files.
INFO Syncing file "exports/my_table2.csv".
INFO Wrote 4 records for table "my_table2".
INFO my_table2: Completed sync (4 rows)
INFO Done syncing.
INFO Loading 4 rows into 'my_table'
INFO COPY my_table_temp ("_sdc_extra", "_sdc_source_bucket", "_sdc_source_file", "_sdc_source_lineno", "classinfo", "id", "userage2", "username") FROM STDIN WITH (FORMAT CSV, ESCAPE '\')
INFO UPDATE 0
INFO INSERT 0 4
INFO Loading 4 rows into 'my_table2'
INFO COPY my_table2_temp ("_sdc_extra", "_sdc_source_bucket", "_sdc_source_file", "_sdc_source_lineno", "classinfo2", "id2", "userage3", "username2") FROM STDIN WITH (FORMAT CSV, ESCAPE '\')
INFO UPDATE 0
INFO INSERT 0 4
{"bookmarks": {"my_table": {"modified_since": "2019-08-22T05:27:52+00:00"}, "my_table2": {"modified_since": "2019-08-22T09:01:45+00:00"}}}
 
 

pg 内容

说明

以上是一个简单的实践，详细的使用可以参考https://github.com/rongfengliang/tap-minio-csv/blob/master/README.md

参考资料

https://github.com/rongfengliang/tap-minio-csv/blob/master/README.md
https://github.com/rongfengliang/tap-minio-csv-demo

singer tap-minio-csv 使用的更多相关文章

  1. python csv 模块的使用

    python csv 模块的使用 歌曲推荐:攀登(live) csv 是用逗号分隔符来分隔列与列之间的. 1. csv的写入 1.简单的写入,一次写入一行 import csv with open(& ...

  2. pipelinewise 基于singer 指南的的数据pipeline 工具

    pipelinewise 是基于开源singer 指南开发的数据pipeline工具,与singer tap 以及target 兼容 支持的特性 内置的elt 特性 轻量级 支持多种复制方法,cdc( ...

  3. Singer 修改tap-s3-csv 支持minio 连接

    singer 团队官方处了一个tap-s3-csv 的tap,对于没有使用aws 的人来说并不是很方便了,所以简单修改了 下源码,可以支持通用的s3 csv 文件的处理,同时发布到了官方pip 仓库中 ...

  4. Singer 学习八 运行&&开发taps、targets (三 开发tap)

    如何没有找到适合的tap,那么我们可以自己开发一个 hello world tap 仅仅是一个程序,我们可以使用任何语言进行编写,根据singer 指南,输出数据到stdout 即可,实际上一个简单的 ...

  5. Singer 学习三 使用Singer进行mongodb 2 postgres 数据转换

    Singer 可以方便的进行数据的etl 处理,我们可以处理的数据可以是api 接口,也可以是数据库数据,或者 是文件 备注: 测试使用docker-compose 运行&&提供数据库 ...

  6. Singer 学习一 使用Singer进行mysql 2 postgres 数据转换

    Singer 因为版本的问题,推荐的运行方式是使用virtualenv,对于taps&& target 的运行都是 推荐使用此方式,不然包兼容的问题太费事了 备注: 使用docker- ...

  7. Supercharging your ETL with Airflow and Singer

    转自:https://www.stitchdata.com/blog/supercharging-etl-with-airflow-and-singer/ singer 团队关于singer 与air ...

  8. 使用singer tap-postgres 同步数据到pg

    singer 是一个很不错的开源etl 解决方案,以下演示一个简单的数据从pg 同步到pg 很简单就是使用tap-postgres + target-postgres 环境准备 对于测试的环境的数据库 ...

  9. [转]Build An Image Manager With NativeScript, Node.js, And The Minio Object Storage Cloud

    本文转自:https://www.thepolyglotdeveloper.com/2017/04/build-image-manager-nativescript-node-js-minio-obj ...

随机推荐

  1. 【题解】PERIOD - Period [POJ1961] [SP263]

    [题解]PERIOD - Period [POJ1961] [SP263] 在进入这道题之前,我们需要了解 kmp 算法 不知道的童鞋可以去看一下Silent_EAG(一个可爱的女孩纸)的讲解. 关于 ...

  2. 命令源码文件——Golang

    源码文件又分为三种,即:命令源码文件.库源码文件和测试源码文件,它们都有着不同的用途和编写规则. 命令源码文件:1.独立程序的入口2.属于main包,包含无参数和无结果的main函数3.main函数执 ...

  3. Oracle的视图和索引

    学习笔记: ##视图     * 概念:视图就是提供一个查询的窗口,所有数据来自于原表.     * 方法:         * 查询语句创建表:             * create table ...

  4. 【开发笔记】-通过js控制input禁止输入空格

    <input type="text" id="fname" onkeyup="myFunction(id)"> <scri ...

  5. JavaScript 之 取消 a 标签的默认行为

    方式一 语法格式: <a href="javascript:;">百度</a> javascript: 是一个伪协议,其他的伪协议还有 mail:  tel ...

  6. CSS 精灵技术（sprite）

    一.精灵技术产生的背景 图所示为网页的请求原理图,当用户访问一个网站时,需要向服务器发送请求,网页上的每张图像都要经过一次请求才能展现给用户.  然而,一个网页中往往会应用很多小的背景图像作为修饰,当 ...

  7. 英文INSURAUNCE保险INSURAUNCE词汇

    世界保险INSURAUNCE 人类社会从开始就面临着自然灾害和意外事故的侵扰,在与大自然抗争的过程中,古代人们就萌生了对付灾害事故的保险思想和原始形态的保险方法.公元前2500年前后,古巴比伦王国国王 ...

  8. Pandas 之 Series / DataFrame 初识

    import numpy as np import pandas as pd Pandas will be a major tool of interest throughout(贯穿) much o ...

  9. Golang: 解析JSON数据之一

    JSON 作为目前最流行的数据传输格式, 相信每个程序员都跟它打过交道吧.使用 Go 语言时,也不可避免的要操作 JSON 数据,令人惊喜的是,Go 内置了序列化和反序列化 JSON 的功能,今天就来 ...

  10. springboot+jndi+tomcat配置多数据源

    1.在application.properties中,添加jndi配置,如下图 2.新建dataSourceConfig类 3.dataSourceConfig类详细代码,这里只贴出其中一个,多个数据 ...