Greenplum 生成加分区语句

在使用greenplum中会使用分区表，但同时分区表需要维护分区；比如加分区，这个过程比较痛苦，查询相关资料以后有了相应的解决办法，但是该办法也不是万能的，有诸多限制，关于限制有兴趣的同学可以查看我文章最后提到的参考资料。

创建2个表（range分区表和list分区表）

CREATE TABLE tb1_partition_list_yyyymmdd (

    id numeric,

    yyyymmdd character varying(128)

)

WITH (appendonly=true, compresslevel=5) DISTRIBUTED BY (id) PARTITION BY LIST(yyyymmdd)

          (

          PARTITION p20120811 VALUES('') WITH (tablename='tb1_partition_list_yyyymmdd_1_prt_p20120811', orientation=row , appendonly=true, compresslevel=5 ),

          PARTITION p20120812 VALUES('') WITH (tablename='tb1_partition_list_yyyymmdd_1_prt_p20120812', orientation=row , appendonly=true, compresslevel=5 )

          );

CREATE TABLE tb1_partition_range_yyyymmdd (

    id numeric,

    yyyymmdd date

)

WITH (appendonly=true, compresslevel=5) DISTRIBUTED BY (id) PARTITION BY RANGE(yyyymmdd)

          (

          PARTITION p20120811 START ('2012-08-11'::date) END ('2012-08-12'::date) WITH (tablename='tb1_partition_range_yyyymmdd_1_prt_p20120811', orientation=row , appendonly=true, compresslevel=5 ),

          PARTITION p20120812 START ('2012-08-12'::date) END ('2012-08-13'::date) WITH (tablename='tb1_partition_range_yyyymmdd_1_prt_p20120812', orientation=row , appendonly=true, compresslevel=5 )

          );

上面分别创建了list分区表和range分区表。

下面创建一个辅助视图：

CREATE VIEW v_pg_add_partitions AS

    SELECT pp.parrelid AS tableoid, prl.parchildrelid, prl.parname AS partitionname, CASE WHEN (pp.parkind = 'h'::"char") THEN 'hash'::text WHEN (pp.parkind = 'r'::"char") THEN 'range'::text WHEN (pp.parkind = '

l'::"char") THEN 'list'::text ELSE NULL::text END AS partitiontype, translate(pg_get_expr(prl.parlistvalues, prl.parchildrelid), '-'':date character varying bpchar numeric double precision timestamp without time

 zone'::text, ''::text) AS partitionlistvalue, "substring"(translate(pg_get_expr(prl.parrangestart, prl.parchildrelid), '-'':date character varying bpchar numeric double precision timestamp without time zone'::t

ext, ''::text), 1, 8) AS partitionrangestart, "substring"(translate(pg_get_expr(prl.parrangeend, prl.parchildrelid), '-'':date character varying bpchar numeric double precision timestamp without time zone'::text

, ''::text), 1, 8) AS partitionrangeend, prl.parruleord AS partitionposition, (("substring"(prl.parlistvalues, 'consttype ([0-9]+)'::text))::integer)::regtype AS listtype, (("substring"(prl.parrangeend, 'constty

pe ([0-9]+)'::text))::integer)::regtype AS rangetype FROM pg_partition pp, pg_partition_rule prl WHERE ((pp.paristemplate = false) AND (prl.paroid = pp.oid));

接下来创建生成添加分区表的命令的函数：

create or replace function add_partition_info(tableoid oid,days_from_now integer)

    returns setof text

as $$

    import datetime

    def now():

        d=datetime.datetime.now()

        format='%Y%m%d'

        return datetime.datetime.strftime(d,format)

    def add_day(d,n):

        format='%Y%m%d'

        d2=datetime.datetime.strptime(d,format)

        d3=d2+datetime.timedelta(days=n)

        return datetime.datetime.strftime(d3,format)

    def add_month(d,n):

        format='%Y%m%d'

        formatymd='%Y%m01'

        if d.__len__() == 6:

            format='%Y%m'

            formatymd='%Y%m'

        d2=datetime.datetime.strptime(d,format)

        d3=d2+datetime.timedelta(days=31*n)

        return datetime.datetime.strftime(d3,formatymd)

    relist=[]

    sql=""" select * from (select *,tableoid::regclass tablename,lead(case when partitionrangeend <> '' then partitionrangeend else partitionlistvalue end) over (partition by tableoid order by partitionposition desc) as pre_value,row_number() over (partition by tableoid order by partitionposition desc) rn from v_pg_add_partitions where substr(partitionname,1,3)='p20' and tableoid=%s) t where rn=1;""" % (tableoid);

    rv=plpy.execute(sql);

    sql_relation="select array_to_string(reloptions,',') reloptions from pg_class where oid=%s" % (tableoid)

    rv_relation=plpy.execute(sql_relation)

    if rv.nrows()!=1:

        return None

    else:

        reloptions = rv_relation[]['reloptions']

        tablename=rv[]['tablename']

        partitiontype=rv[]['partitiontype']

        partitionname=rv[]['partitionname']

        pre_value=rv[]['pre_value']

        now_add_7days=add_day(now(),days_from_now)

        if partitiontype=='range':

            rangetype=rv[]['rangetype']

            partitionrangestart=rv[]['partitionrangestart']

            partitionrangeend=rv[]['partitionrangeend']

            interval = int(partitionrangeend) - int(pre_value)

            if partitionname.__len__()==7:

                func_add=add_month

                interval=int(partitionrangeend[0:6]) - int(pre_value[0:6])

            elif partitionname.__len__()==9:

                func_add=add_day

            else:

                return None

            while partitionrangestart < now_add_7days:

                partitionrangestart = func_add(partitionrangestart,interval)

                partitionrangeend = func_add(partitionrangeend,interval)

                partitionname="p"+func_add(partitionname[1:],interval)

                add_sql = "alter table %s add partition %s start ('%s'::%s) end ('%s'::%s) " % (tablename,partitionname,partitionrangestart,rangetype,partitionrangeend,rangetype)

                if reloptions != None and reloptions != '':

                    add_sql+='with (%s);' % (reloptions)

                else:

                    add_sql+=";"

                relist.append(add_sql)

        if partitiontype=='list':

            listtype=rv[]['listtype']

            partitionlistvalue=rv[]['partitionlistvalue']

            interval=int(partitionlistvalue) - int(pre_value)

            if partitionname.__len__()==7:

                func_add = add_month

            elif partitionname.__len__()==9:

                func_add=add_day

            else:

                return None

            while partitionlistvalue < now_add_7days:

                partitionlistvalue=func_add(partitionlistvalue,interval)

                partitionname="p"+func_add(partitionname[1:],interval)

                add_sql="alter table %s add partition %s values ('%s'::%s)" % (tablename,partitionname,partitionlistvalue,listtype)

                if reloptions != None and reloptions != '':

                    add_sql +='with (%s);' % (reloptions)

                else:

                    add_sql+=";"

                relist.append(add_sql)

        return relist

$$ LANGUAGE plpythonu;

最后我们进行测试（增加当前时间3天内的分区）

test_db=# SELECT add_partition_info('tb1_partition_range_yyyymmdd'::regclass,3);

                                                                    add_partition_info

----------------------------------------------------------------------------------------------------------------------------------------------------------

 alter table tb1_partition_range_yyyymmdd add partition p20120813 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120814 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120815 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120816 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120817 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120818 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120819 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120820 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120821 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120822 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120823 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120824 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120825 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120826 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120827 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120828 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120829 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120830 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120831 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120901 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120902 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120903 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120904 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120905 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120906 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120907 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120908 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

 alter table tb1_partition_range_yyyymmdd add partition p20120909 start (''::date) end (''::date) with (appendonly=true,compresslevel=5);

再创建一个range分区且是按月分区的表

CREATE TABLE tb1(

    id numeric,

    yyyymmdd date

)DISTRIBUTED BY (id) PARTITION BY RANGE(yyyymmdd)

          (

          PARTITION p201208 START ('2012-08-01'::date) END ('2012-09-01'::date) ,

          PARTITION p201209 START ('2012-09-01'::date) END ('2012-10-01'::date)

          );

test_db=# SELECT add_partition_info('tb1'::regclass,1);

                                   add_partition_info

-----------------------------------------------------------------------------------------

 alter table tb1 add partition p201210 start (''::date) end (''::date) ;

 alter table tb1 add partition p201211 start (''::date) end (''::date) ;

 alter table tb1 add partition p201212 start (''::date) end (''::date) ;

 alter table tb1 add partition p201301 start (''::date) end (''::date) ;

 alter table tb1 add partition p201302 start (''::date) end (''::date) ;

 alter table tb1 add partition p201303 start (''::date) end (''::date) ;

 alter table tb1 add partition p201304 start (''::date) end (''::date) ;

 alter table tb1 add partition p201305 start (''::date) end (''::date) ;

 alter table tb1 add partition p201306 start (''::date) end (''::date) ;

 alter table tb1 add partition p201307 start (''::date) end (''::date) ;

 alter table tb1 add partition p201308 start (''::date) end (''::date) ;

 alter table tb1 add partition p201309 start (''::date) end (''::date) ;

 alter table tb1 add partition p201310 start (''::date) end (''::date) ;

 alter table tb1 add partition p201311 start (''::date) end (''::date) ;

 alter table tb1 add partition p201312 start (''::date) end (''::date) ;

 alter table tb1 add partition p201401 start (''::date) end (''::date) ;

 alter table tb1 add partition p201402 start (''::date) end (''::date) ;

 alter table tb1 add partition p201403 start (''::date) end (''::date) ;

 alter table tb1 add partition p201404 start (''::date) end (''::date) ;

 alter table tb1 add partition p201405 start (''::date) end (''::date) ;

 alter table tb1 add partition p201406 start (''::date) end (''::date) ;

 alter table tb1 add partition p201407 start (''::date) end (''::date) ;

 alter table tb1 add partition p201408 start (''::date) end (''::date) ;

 alter table tb1 add partition p201409 start (''::date) end (''::date) ;

 alter table tb1 add partition p201410 start (''::date) end (''::date) ;

 alter table tb1 add partition p201411 start (''::date) end (''::date) ;

 alter table tb1 add partition p201412 start (''::date) end (''::date) ;

可以看见也是没有问题的。

参考资料：

《greenplum企业应用实战》

Greenplum 生成加分区语句的更多相关文章

sqlserver 表中记录生成insert，可以加条件，可以生成建表语句
sqlserver 表中记录生成insert,可以加条件,可以生成建表语句 create PROCEDURE [sp_getinsert] ( ) , --如果非默认架构,可以加上架构名例如:sch ...
GreenPlum数据加载
1. copy命令对于数据加载,GreenPlum数据库提供copy工具,copy工具源于PostgreSQL数据库,copy命令支持文件与表之间的数据加载和表对文件的数据卸载.使用copy命令进行 ...
phalcon Model 'partitions' could not be loaded(模型不支持分区语句)
注意: 很明确提示用phalcon自带的模型层是不能用partition这个关键字的解决方法: 自己写个PDO类然后用pdo中的query方法执行语句成功: mysql分区目的是减少数据库的负担 ...
h5 录音自动生成proto Js语句 UglifyJS-- 对你的js做了什么【原码笔记】-- protobuf.js 与 Long.js 【微信开发】-- 发送模板消息能编程与会编程 vue2入坑随记(二) -- 自定义动态组件微信上传图片
得益于前辈的分享,做了一个h5录音的demo.效果图如下: 点击开始录音会先弹出确认框: 首次确认允许后,再次录音不需要再确认,但如果用户点击禁止,则无法录音: 点击发送将录音内容发送到对话框中.点 ...
快速将一个表的数据生成SQL插入语句
将一个表中的数据生成SQL插入语句,方便系统快速初始化,在数据库中执行创建以下过程就可以了. ) Drop Procedure GenerateData go CREATE PROCEDURE Gen ...
这种代码怎么改写？以致于在下次增加CustomsType时，不需要再加 if 语句。
最近看到项目里一段代码如下: excelObject excel = new excelObject(); if (loadbill.CustomsType == 1) excel.IDownload ...
使用EntityFramework6.1的DbCommandInterceptor拦截生成的SQL语句
开始 EF6.1也出来不少日子了,6.1相比6.0有个很大的特点就是新增了System.Data.Entity.Infrastructure.Interception 命名空间,此命名空间下的对象可以 ...
用LinqPad查看Nhibernate生成的sql语句
使用Nhibernate开发一般都要对Nhibernate生成的sql语句进行查看及分析,查看Nhibernate生成的sql语句,可以使用NHProfiler和log4net.但NHProfiler ...
【转】Hibernate利用@DynamicInsert和@DynamicUpdate生成动态SQL语句
原文链接:http://www.cnblogs.com/quanyongan/p/3152290.html 最近在使用Hibernate4中,发现两个很有奥秘的注解 @DynamicInsert 和 ...

随机推荐

使用Hue上传hive数据
大概逻辑是先上传hdfs数据,然后创建hive外部表,关联到hdfs上传数据的位置. 截图比较概要,但是用起来很简单 1.创建路径和上传文件 2.创建外部表
JMeter正则表达式-学习（3）
同时关联多个值的方法: { : ", : "results": : [ : : { : : : "total_earnings":"&quo ...
Liferay 6.2 改造系列之二十：修改默认添加用户权限及用户是否拥有个人站点的配置
1.修改默认添加用户权限为User,而非Powers User: 在/portal-master/portal-impl/src/portal.properties文件中有如下配置: # # Inpu ...
Spring事务解析1-使用介绍
spring的事务控制让我们从复杂的事务处理中得到解脱,是我们再也不需要去处理获得连接,关闭连接,事务提交和回滚等操作,再也不需要在事务相关的方法中处理大量的try..catch...finally代 ...
PHP学习之字符串
1.字符串变量用于存储并处理文本: 2.字符串变量用于包含有字符的值,在创建字符串后,就可以对它进行操作了,可以直接在函数中使用字符串,或把它存储在变量中: <?php $txt="H ...
IconFont和FontAwesome的区别？
一.[Iconfont] Iconfont支持所有低版本浏览器: Iconfont的图标库更大: Iconfont可以用自己上传的svg,但是要花费大量时间和耐心去设计AI图标: Iconfont的使 ...
Java命令行输入求和的简单学习
思想:命令行输入的参数,必须先转换为数字才能进行加法计算,这就需要引用java.util.Scanner; 流程框图: 源代码: //实现几个整数相加的程序 //高开拓,2015.9.26 packa ...
利用Hive实现求两条相邻数据时间差
1.Hive row_number() 函数的高级用法 row_num 按照某个字段分区显示第几条数据 select imei,ts,fuel_instant,gps_longitude,gps_la ...
MFC 修改单文档 SDI 窗体标题
程序主窗口标题栏的一般形式为: 文档标题—主窗口标题 1.修改文档标题修改CExamDoc::onNewDocument()函数为: BOOL CExamDoc::OnNewDocument() { ...
data.table包
data.table 1.生成一个data.table对象生成一个data.table对象,记为DT. library(data.table) :],V3=round(rnorm(),),V4=:) ...

Greenplum 生成加分区语句

Greenplum 生成加分区语句的更多相关文章

随机推荐

热门专题