dbt 基本试用

dbt 是一个很不错的进行etl 中的t 处理的工具，灵活简单，我们需要写的就是select 语句
dbt 帮助我们进行处理

测试集成了graphql 以及使用docker 运行

安装

pip install dbt

创建简单demo

• 使用dbt 处理

dbt init demoapp

• 配置profile（pg 数据库）

~/.dbt/profiles.yml
default:
  target: dev
  outputs:
    dev:
      type: postgres
      host: 127.0.0.1
      user: root
      pass: password
      port: 5432
      dbname: dbt
      schema: dbt
      threads: 3
pg:
  target: dev
  outputs:
    dev:
      type: postgres
      host: 127.0.0.1
      user: root
      pass: password
      port: 5433
      dbname: dbt
      schema: dbt

• graphql 集成

  docker-compose 文件

version: "3"
services:
   graphql-engine:
    image: hasura/graphql-engine:v1.0.0-alpha24
    ports:
    - "8080:8080"
    depends_on:
    - database
    environment:
    - "POSTGRES_PASSWORD:password"
    command: >
      /bin/sh -c "
      graphql-engine --database-url postgres://root:password@database:5432/dbt serve --enable-console;
      "
   database:
    image: postgres
    environment:
      POSTGRES_USER: "root"
      POSTGRES_PASSWORD: "password"
      POSTGRES_DB: "dbt"
    ports:
      - "5432:5432"
   database2:
    image: postgres
    environment:
      POSTGRES_USER: "root"
      POSTGRES_PASSWORD: "password"
      POSTGRES_DB: "dbt"
    ports:
      - "5433:5432"

• 修改简单model

models/example/my_first_dbt_model.sql
-- Welcome to your first dbt model!
-- Did you know that you can also configure models directly within
-- the SQL file? This will override configurations stated in dbt_project.yml

-- Try changing 'view' to 'table', then re-running dbt
{{ config(materialized='view') }}

select 1 as id, 'v1' as version,'default' as v,'default' as v2
union all
select 2,'v2','default','default'
union all
select 3,'v3','default','default'
union all
select 4,'v4','default','default'
union all
select 5,'v5','default','default'
union all
select 6,'v6','default','default'
union all
select 7,'v7','default','default'
union all
select 8,'v8','default','default'
union all
select 9,'v9','default','default'

运行

• docker-compose 启动

docker-compose up -d

• 运行dbt

dbt fun

• 效果

• 支持graphql 配置

  配置即可

• 数据查询

参考资料

https://github.com/rongfengliang/dbt-demo
https://www.getdbt.com/