dbt 0.14.0 试用

dbt 0.14.0 在最近已经发布了，dbt server 的还是很不错的特性，以下安装试用下几个新功能

环境准备

安装

如果没有安装的：

pip install  dbt

已经安装的：

pip install -U dbt

pg 数据库环境准备

使用docker-compose 运行

• docker-compose 文件

version: "3"

services:

   database:

    image: postgres

    environment:

      POSTGRES_USER: "postgres"

      POSTGRES_PASSWORD: "dalong"

    ports:

      - "5432:5432"

   database2:

    image: postgres

    environment:

      POSTGRES_USER: "postgres"

      POSTGRES_PASSWORD: "dalong"

    ports:

      - "5433:5432"

rpc

创建简单项目

• dbt init

dbt init 14demo

• 配置profile

  profiles.yml 文件

# For more information on how to configure this file, please see:

# https://github.com/fishtown-analytics/dbt/blob/master/sample.profiles.yml

default:

  target: dev

  outputs:

    dev:

      type: postgres

      host: 127.0.0.1

      user: postgres

      pass: password

      port: 5432

      dbname: postgres

      schema: dalong

      threads: 3

pg:

  target: dev

  outputs:

    dev:

      type: postgres

      host: 127.0.0.1

      user: postgres

      pass: dalong

      port: 5432

      dbname: postgres

      schema: public

      threads: 3

demo:

  target: dev

  outputs:

    dev:

      type: postgres

      host: 127.0.0.1

      user: postgres

      pass: dalong

      port: 5432

      dbname: postgres

      schema: demo

      threads: 3

基本使用

• 启动pg

docker-compose up -d

• 启动dbt server rpc 服务

dbt rpc

效果

dbt rpc 

Running with dbt=0.14.0

Found 116 macros, 0 analyses, 0 sources, 0 seed files, 0 snapshots, 0 tests, 1 models, 0 operations

​

14:39:08 | Concurrency: 3 threads (target='dev')

14:39:08 | 

14:39:08 | Done.

Serving RPC server at 0.0.0.0:8580

Supported methods: ['compile', 'run', 'ps', 'kill']

Send requests to http://localhost:8580/jsonrpc

 

• rpc 服务请求格式

{

    "jsonrpc": "2.0",

    "method": "{ a valid rpc server command }",

    "id": "{ a unique identifier for this query }",

    "params": {

        "timeout": { timeout for the query in seconds },

        "sql": "{ base64 encoded dbt-sql query }",

        "name": "{ an identifying name for this query }"

    }

}

• 一个rpc 服务参考例子

{

    "jsonrpc": "2.0",

    "method": "compile",

    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",

    "params": {

        "timeout": 60,

        "sql": "c2VsZWN0IHt7IDEgKyAxIH19IGFzIGlk",

        "name": "my_first_query"

    }

}

 

curl 格式

curl -X GET \

  http://localhost:8580/jsonrpc \

  -H 'Accept: */*' \

  -H 'Cache-Control: no-cache' \

  -H 'Connection: keep-alive' \

  -H 'Content-Type: text/plain' \

  -H 'Host: localhost:8580' \

  -H 'accept-encoding: gzip, deflate' \

  -H 'cache-control: no-cache' \

  -H 'content-length: 229' \

  -H 'cookie: Cookie_6=value; platform=order' \

  -b 'Cookie_6=value; platform=order' \

  -d '{

    "jsonrpc": "2.0",

    "method": "compile",

    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",

    "params": {

        "timeout": 60,

        "sql": "c2VsZWN0IHt7IDEgKyAxIH19IGFzIGlk",

        "name": "my_first_query"

    }

}'

 

返回数据

{

    "jsonrpc": "2.0",

    "result": {

        "timing": [

            {

                "completed_at": "2019-07-11T06:55:53.542902Z",

                "started_at": "2019-07-11T06:55:53.531315Z",

                "name": "compile"

            },

            {

                "completed_at": "2019-07-11T06:55:53.543440Z",

                "started_at": "2019-07-11T06:55:53.543151Z",

                "name": "execute"

            }

        ],

        "raw_sql": "select {{ 1 + 1 }} as id",

        "compiled_sql": "select 2 as id",

        "logs": [

            {

                "timestamp": "2019-07-11 14:55:53,428",

                "message": "Parsing rpc.my_new_package.my_first_query",

                "levelname": "DEBUG",

                "level": 10

            },

            {

                "timestamp": "2019-07-11 14:55:53,432",

                "message": "Acquiring new postgres connection \"my_first_query\".",

                "levelname": "DEBUG",

                "level": 10

            },

            {

                "timestamp": "2019-07-11 14:55:53,433",

                "message": "Opening a new connection, currently in state init",

                "levelname": "DEBUG",

                "level": 10

            },

            {

                "timestamp": "2019-07-11 14:55:53,522",

                "message": "Found 116 macros, 0 analyses, 0 sources, 1 None, 0 seed files, 0 snapshots, 0 tests, 1 models, 0 operations",

                "levelname": "NOTICE",

                "level": 15

            },

            {

                "timestamp": "2019-07-11 14:55:53,523",

                "message": "Acquiring new postgres connection \"my_first_query\".",

                "levelname": "DEBUG",

                "level": 10

            },

            {

                "timestamp": "2019-07-11 14:55:53,524",

                "message": "Opening a new connection, currently in state init",

                "levelname": "DEBUG",

                "level": 10

            },

            {

                "timestamp": "2019-07-11 14:55:53,531",

                "message": "Compiling rpc.my_new_package.my_first_query",

                "levelname": "DEBUG",

                "level": 10

            }

        ]

    },

    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d"

}

ls 命令

dbt ls

效果：

dbt ls

my_new_package.example.my_first_dbt_model

说明

dbt 0.14.0 的好多新特性还是很不错的

参考资料

https://docs.getdbt.com/docs/rpc
https://docs.getdbt.com/v0.14/docs/run-operation
https://docs.getdbt.com/docs/list
https://github.com/rongfengliang/dbt-0.14.0-learning