PostgreSQL Partitions

why we need partitions

The first and most demanding reason to use partitions in a database is to increase the performance of the database. This is achieved by partition-wise joins; if a user’s queries perform a lot of full-table scans, partitioning will help vastly, because partitions will limit the scope of this search. The second important reason to partition is ease of managing large tables.

Partitioning always helps manage large objects. Although it is applicable to objects of any size, the advantages are more apparent in large tables. When a user recreates an index on a nonpartitioned table, the only option is to build the entire index in one statement. On the other hand, if the table is partitioned, the user can rebuild partitions of the local indexes one at a time.

PostgreSQL supports table partitioning through table inheritance, which means every

partition will be created as a child table of a single parent table. Partitioning is performed in such a way that every child table inherits a single parent table. The parent table will be empty; it exists just to describe the whole dataset. Currently in PostgreSQL, partitioning can be implemented in range partitioning or list partitioning.

Range partitioning can be done, for example, by record number ranges (such as record 0 to 100 or 100 to 200) or even using date ranges (such as from 2014-11-01 to 2014-11-30).

List partitioning can be done, for example, by a list of cities (such as New York, Boston,

Chicago, and Houston) or list of departments (such as HR, finance, administration, and so on).

There are five simple steps used to create a partition in PostgreSQL, which are as follows:

1. Create the master table.

2. Create multiple child tables without having an overlapped table constraint.

3. Create indexes.

4. Create a trigger function to insert data into child tables.

5. Enable the constraint exclusion.

Range partition

The range partition is the partition in which we partition a table into ranges defined by a single column or multiple columns. When defining the ranges, the user will have to take care that ranges should be connected and not overlapping with each other; moreover, ranges must be defined using the < value operator. For instance, one can partition by date ranges or ranges of identifiers for specific business objects.

Creating the master table

[postgres@minion1 bin]$ ./psql

psql (9.4.5)

Type "help" for help.

postgres=# create database partition_db;

CREATE DATABASE

postgres=# \c partition_db

You are now connected to database "partition_db" as user "postgres".

partition_db=# create database warehouse_db;

CREATE DATABASE

partition_db=# \c warehouse_db

You are now connected to database "warehouse_db" as user "postgres".

warehouse_db=# CREATE TABLE sales_record

warehouse_db-# (

warehouse_db(# id NUMERIC PRIMARY KEY,

warehouse_db(# sales_amount NUMERIC,

warehouse_db(# sales_date DATE NOT NULL DEFAULT CURRENT_DATE

warehouse_db(# );

CREATE TABLE

Creating a range partition table

Each partition will have two months’ data.

Create the sales_record_m1_to_m2 child table as follows:

warehouse_db=# CREATE TABLE sales_record_m1_to_m2

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, sales_date),

warehouse_db(# CHECK (sales_date >= DATE '2015-01-01'

warehouse_db(# AND sales_date < DATE '2015-03-01')

warehouse_db(# )

warehouse_db-# INHERITS (sales_record);

CREATE TABLE

This child table will contain data of January and February.

Now, create the sales_record_m3_to_m4 child table as follows:

warehouse_db=# CREATE TABLE sales_record_m3_to_m4

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, sales_date),

warehouse_db(# CHECK (sales_date >= DATE '2015-03-01'

warehouse_db(# AND sales_date < DATE '2015-05-01')

warehouse_db(# )

warehouse_db-# INHERITS (sales_record);

CREATE TABLE

This child table will contain data of March and April.

Create the sales_record_m5_to_m6 child table as follows:

warehouse_db=# CREATE TABLE sales_record_m5_to_m6

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, sales_date),

warehouse_db(# CHECK (sales_date >= DATE '2015-05-01'

warehouse_db(# AND sales_date < DATE '2015-07-01')

warehouse_db(# )

warehouse_db-# INHERITS (sales_record);

CREATE TABLE

This child table will contain data of May and June.

Create the sales_record_m7_to_m8 child table as follows:

warehouse_db=# CREATE TABLE sales_record_m7_to_m8

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, sales_date),

warehouse_db(# CHECK (sales_date >= DATE '2015-07-01'

warehouse_db(# AND sales_date < DATE '2015-09-01')

warehouse_db(# )

warehouse_db-# INHERITS (sales_record);

CREATE TABLE

This child table will contain data of July and August.

Create the sales_record_m9_to_m10 child table as follows:

warehouse_db=# CREATE TABLE sales_record_m9_to_m10

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, sales_date),

warehouse_db(# CHECK (sales_date >= DATE '2015-09-01'

warehouse_db(# AND sales_date < DATE '2015-11-01')

warehouse_db(# )

warehouse_db-# INHERITS (sales_record);

CREATE TABLE

This child table will contain data of September and October.

Now, create the sales_record_m11_to_m12 child table as follows:

warehouse_db=# CREATE TABLE sales_record_m11_to_m12

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, sales_date),

warehouse_db(# CHECK (sales_date >= DATE '2015-11-01'

warehouse_db(# AND sales_date < DATE '2016-01-01')

warehouse_db(# )

warehouse_db-# INHERITS (sales_record);

CREATE TABLE

This child table will contain data of November and December.

Verifing that the tables are linked and the partition is successfully created using the following query:

warehouse_db=# \d+ sales_record

Table "public.sales_record"

Column    |  Type   |              Modifiers               | Storage | Stats target | Description

--------------+---------+--------------------------------------+---------+--------------+-------------

id           | numeric | not null                             | main    |              |

sales_amount | numeric |                                      | main    |              |

sales_date   | date    | not null default ('now'::text)::date | plain   |              |

Indexes:

"sales_record_pkey" PRIMARY KEY, btree (id)

Child tables: sales_record_m11_to_m12,

sales_record_m1_to_m2,

sales_record_m3_to_m4,

sales_record_m5_to_m6,

sales_record_m7_to_m8,

sales_record_m9_to_m10

Creating an index on child tables

Now, we are going to create indexes on child tables to speed up the sales_day field usage, using almost all queries (INSERT, SELECT, and UPDATE) on the date field.

Create the m1_to_m2_sales_date index on the sales_record_m1_to_m2 child table as follows:

warehouse_db=# CREATE INDEX m1_to_m2_sales_date ON sales_record_m1_to_m2

warehouse_db-# (sales_date);

CREATE INDEX

Create the m3_to_m4_sales_date index on the sales_record_m3_to_m4 child table as

follows:

warehouse_db=# CREATE INDEX m3_to_m4_sales_date ON sales_record_m3_to_m4

warehouse_db-# (sales_date);

CREATE INDEX

Now, let’s create the m5_to_m6_sales_date index on the sales_record_m5_to_m6 child table:

warehouse_db=# CREATE INDEX m5_to_m6_sales_date ON sales_record_m5_to_m6

warehouse_db-# (sales_date);

CREATE INDEX

Create the m7_to_m8_sales_date index on the sales_record_m7_to_m8 child table as

follows:

warehouse_db=# CREATE INDEX m7_to_m8_sales_date ON sales_record_m7_to_m8

warehouse_db-# (sales_date);

CREATE INDEX

Then, create the m9_to_m10_sales_date index on the sales_record_m9_to_m10 child table as follows:

warehouse_db=# CREATE INDEX m9_to_m10_sales_date ON sales_record_m9_to_m10

warehouse_db-# (sales_date);

CREATE INDEX

Create the m11_to_m12_sales_date index on the sales_record_m11_to_m12 child table as follows:

warehouse_db=# CREATE INDEX m11_to_m12_sales_date ON

warehouse_db-# sales_record_m11_to_m12 (sales_date);

CREATE INDEX

Creating a trigger on the master table

Conditions must be exactly the same as what the child tables check.

creating a trigger function using the following syntax:

warehouse_db=# CREATE OR REPLACE FUNCTION sales_record_insert()

warehouse_db-# RETURNS TRIGGER AS $$

warehouse_db$# BEGIN

warehouse_db$# IF (NEW.sales_date >= DATE '2015-01-01' AND NEW.sales_date < DATE '2015-03-01') THEN

warehouse_db$# INSERT INTO sales_record_m1_to_m2 VALUES (NEW.*);

warehouse_db$# ELSEIF (NEW.sales_date >= DATE '2015-03-01' AND NEW.sales_date < DATE '2015-05-01') THEN

warehouse_db$# INSERT INTO sales_record_m3_to_m4 VALUES (NEW.*);

warehouse_db$# ELSEIF (NEW.sales_date >= DATE '2015-05-01' AND NEW.sales_date < DATE '2015-07-01') THEN

warehouse_db$# INSERT INTO sales_record_m5_to_m6 VALUES (NEW.*);

warehouse_db$# ELSEIF (NEW.sales_date >= DATE '2015-07-01' AND NEW.sales_date < DATE '2015-09-01') THEN

warehouse_db$# INSERT INTO sales_record_m7_to_m8 VALUES (NEW.*);

warehouse_db$# ELSEIF (NEW.sales_date >= DATE '2015-09-01' AND NEW.sales_date < DATE '2015-11-01') THEN

warehouse_db$# INSERT INTO sales_record_m9_to_m10 VALUES (NEW.*);

warehouse_db$# ELSEIF (NEW.sales_date >= DATE '2015-11-01' AND NEW.sales_date < DATE '2016-01-01') THEN

warehouse_db$# INSERT INTO sales_record_m11_to_m12 VALUES (NEW.*);

warehouse_db$# ELSE

warehouse_db$# RAISE EXCEPTION 'Date is out of range. Something is wrong with sales_record_insert_trigger() function';

warehouse_db$# END IF;

warehouse_db$# RETURN NULL;

warehouse_db$# END;

warehouse_db$# $$LANGUAGE plpgsql;

CREATE FUNCTION

This function will simply populate the data in the respective child table on the basis of the sales_date condition.

Now, the supportive trigger will call the preceding trigger function automatically

whenever a user uses the INSERT, UPDATE, or DELETE operations on data in the master table. Creating the supportive trigger in the following manner:

warehouse_db=# CREATE TRIGGER sales_day_trigger

warehouse_db-# BEFORE INSERT ON sales_record

warehouse_db-# FOR EACH ROW

warehouse_db-# EXECUTE PROCEDURE sales_record_insert();

CREATE TRIGGER

Enabling the constraint exclusion

The final step in implementation of partitioning is enabling the constraint exclusion.Constraint exclusion is backed by the CHECK constraints that we have used in our child table’s CREATE syntax. Do remember that if the constraint exclusion is disabled, then our

queries will not use CHECK constraints, and as a result, every query scan will be done on all the child tables and thus will reduce performance; this is why constraint exclusion to is very important when using partitioned tables.

The steps to enable constraint exclusion are as follows:

1. Open the postgresql.conf file that is present in the data directory on your default installation path. In most cases, it is /usr/local/pgsql/data.

2. Set constraint exclusion on with the following row in postgresql.conf:

constraint_exclusion = on

Alternatively, you can set constraint exclusion to on using the following command on

psql:

warehouse_db=# SET constraint_exclusion = on;

Congratulations! Finally, the master table is available for the DML and DDL operations,and all the INSERT, SELECT and DELETE operations go to the child tables by date.

Performing DML operations on a partition table

When a user inserts a row in the master table, our trigger sales_day_trigger will be triggered, and it will call our sales_record_insert() trigger function, and based on sales_date, the insertion will be made to a specific child table. Now, let’s insert a few records into the sales_record table.We will then perform the SELECT statements to verify that records are properly populated in child tables. This can be done in the following manner:

warehouse_db=# INSERT INTO sales_record

(id, sales_amount, sales_date)

VALUES

(1, 500, TO_DATE('02/12/2015','MM/DD/YYYY'));

INSERT 0 0

warehouse_db=# INSERT INTO sales_record

warehouse_db-# (id, sales_amount, sales_date)

warehouse_db-# VALUES

warehouse_db-# (2, 1500, TO_DATE('03/10/2015','MM/DD/YYYY'));

INSERT 0 0

warehouse_db=# INSERT INTO sales_record

warehouse_db-# (id, sales_amount, sales_date)

warehouse_db-# VALUES

warehouse_db-# (3, 2500, TO_DATE('05/15/2015','MM/DD/YYYY'));

INSERT 0 0

warehouse_db=# INSERT INTO sales_record

warehouse_db-# (id, sales_amount, sales_date)

warehouse_db-# VALUES

warehouse_db-# (4, 2000, TO_DATE('07/25/2015','MM/DD/YYYY'));

INSERT 0 0

warehouse_db=# INSERT INTO sales_record

warehouse_db-# (id, sales_amount, sales_date)

warehouse_db-# VALUES

warehouse_db-# (5, 2200, TO_DATE('09/15/2015','MM/DD/YYYY'));

INSERT 0 0

warehouse_db=# INSERT INTO sales_record

warehouse_db-# (id, sales_amount, sales_date)

warehouse_db-# VALUES

warehouse_db-# (6, 1200, TO_DATE('11/15/2015','MM/DD/YYYY'));

INSERT 0 0

We have inserted six records, and we are now going to perform the SELECT queries on our child tables to verify that our child tables get the right data. Let’s do a SELECT operation on the sales_record_m3_to_m4 table in the following manner:

warehouse_db=# SELECT * FROM sales_record_m3_to_m4;

id | sales_amount | sales_date

----+--------------+------------

2 |         1500 | 2015-03-10

(1 row)

Let’s do a select operation on the sales_record_m9_to_m10 table in the following

manner:

warehouse_db=# SELECT * FROM sales_record_m9_to_m10;

id | sales_amount | sales_date

----+--------------+------------

5 |         2200 | 2015-09-15

(1 row)

As you can see in the preceding result, the sales_record_m9_to_m10 child table contains the record that has sales_date for September and October.

A SELECT operation on the parent/master table will grab data from all child tables. This can be seen using the following statement:

warehouse_db=# SELECT * FROM sales_record;

id | sales_amount | sales_date

----+--------------+------------

1 |          500 | 2015-02-12

2 |         1500 | 2015-03-10

3 |         2500 | 2015-05-15

4 |         2000 | 2015-07-25

5 |         2200 | 2015-09-15

6 |         1200 | 2015-11-15

(6 rows)

Handling the UPDATE and DELETE statements on a partition table

We don’t need any UPDATE or DELETE triggers, the INSERT trigger is self-sufficient to handle the UPDATE or DELETE statement as well. First, let’s perform an UPDATE action and verify the impact on the child table in the following manner:

warehouse_db=# UPDATE sales_record SET sales_date='2015-9-13' WHERE id = 5;

UPDATE 1

The preceding query will update the record of the date 2015-9-15 with id = 5.

Let’s perform a SELECT operation to verify the update of the record in the following

manner:

warehouse_db=# SELECT * FROM sales_record_m9_to_m10;

id | sales_amount | sales_date

----+--------------+------------

5 |         2200 | 2015-09-13

(1 row)

We can now verify that the record is updated and sales_date is changed to a new date,that is, 2015-09-13.

An update on a table that can cause row movement between children requires an UPDATE trigger.

Now, let’s perform a simple DELETE operation and verify the impact on partitioned tables:

warehouse_db=# DELETE FROM sales_record where sales_date = '2015-9-13';

DELETE 1

warehouse_db=# SELECT * FROM sales_record_m9_to_m10;

id | sales_amount | sales_date

----+--------------+------------

(0 rows)

Since the SELECT statement did not return any record, this confirms that we have successfully deleted the record with sales_date = '2015-9-13'.

List partition

List partition is very much similar to range partition. The table is partitioned by explicitly listing which key values appear in each partition. In list partition, each partition is defined and designated based on a column value in one set of value lists, instead of one set of adjoining ranges of values. This will be done by defining each partition by means of the values IN (value_list) syntax, where value_list is a comma-separated list of values.

We will create a master table that will have a sales record along with the city information. The list partition will use the city column as a base to create the child partitions. Let’s create a master table first in the following manner:

warehouse_db=# CREATE TABLE sales_record_listpart

warehouse_db-# (

warehouse_db(# id NUMERIC primary key,

warehouse_db(# sales_date date,

warehouse_db(# sales_amount NUMERIC,

warehouse_db(# city text

warehouse_db(# );

CREATE TABLE

Now, let’s create the child tables on the basis of the city list.

Create the sales_record_list1 table in the following manner:

warehouse_db=# CREATE TABLE sales_record_list1

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, city),

warehouse_db(# CHECK (city IN ('new york', 'sydney'))

warehouse_db(# )

warehouse_db-# INHERITS (sales_record_listpart);

CREATE TABLE

Now, create the sales_record_list2 table in the following manner:

warehouse_db=# CREATE TABLE sales_record_list2

warehouse_db-# (

warehouse_db(# PRIMARY KEY (id, city),

warehouse_db(# CHECK (city IN ('Islamabad', 'Boston', 'London'))

warehouse_db(# )

warehouse_db-# INHERITS (sales_record_listpart);

CREATE TABLE

Let’s create the index for the sales_record_list1 table:

warehouse_db=# CREATE INDEX list1_index ON sales_record_list1(city);

CREATE INDEX

Let’s create the index for the sales_record_list2 table:

warehouse_db=# CREATE INDEX list2_index ON sales_record_list2 (city);

CREATE INDEX

Now, create the trigger function in the following manner:

warehouse_db=# CREATE OR REPLACE FUNCTION sales_record_list_insert()

warehouse_db-# RETURNS TRIGGER AS $$

warehouse_db$# BEGIN

warehouse_db$# IF (NEW.city IN ('new york', 'sydney')) THEN

warehouse_db$# INSERT INTO sales_record_list1 VALUES (NEW.*);

warehouse_db$# ELSEIF (NEW.city IN ('Islamabad', 'Boston', 'London')) THEN

warehouse_db$# INSERT INTO sales_record_list2 VALUES (NEW.*);

warehouse_db$# ELSE

warehouse_db$# RAISE EXCEPTION 'CITY not present in this lists';

warehouse_db$# END IF;

warehouse_db$# RETURN NULL;

warehouse_db$# END;

warehouse_db$# $$LANGUAGE plpgsql;

CREATE FUNCTION

In the end, we need to create the supporting trigger in the following manner:

warehouse_db=# CREATE TRIGGER sales_day_trigger

warehouse_db-# BEFORE INSERT ON sales_record_listpart

warehouse_db-# FOR EACH ROW

warehouse_db-# EXECUTE PROCEDURE sales_record_list_insert();

CREATE TRIGGER

Verifing that the partition is linked with the master table using the following

command:

warehouse_db=# \d+ sales_record_listpart

Table "public.sales_record_listpart"

Column    |  Type   | Modifiers | Storage  | Stats target | Description

--------------+---------+-----------+----------+--------------+-------------

id           | numeric | not null  | main     |              |

sales_date   | date    |           | plain    |              |

sales_amount | numeric |           | main     |              |

city         | text    |           | extended |              |

Indexes:

"sales_record_listpart_pkey" PRIMARY KEY, btree (id)

Triggers:

sales_day_trigger BEFORE INSERT ON sales_record_listpart FOR EACH ROW EXECUTE PROCEDURE sales_record_list_insert()

Child tables: sales_record_list1,

sales_record_list2

Now, let’s do some quick inserts and verify that our list partition is also working how we expect it do so:

warehouse_db=# INSERT INTO sales_record_listpart

warehouse_db-# (id, sales_date, sales_amount, city)

warehouse_db-# VALUES

warehouse_db-# (1,'15-APR-2008',1200,'sydney');

INSERT 0 0

warehouse_db=# INSERT INTO sales_record_listpart

warehouse_db-# (id, sales_date, sales_amount, city)

warehouse_db-# VALUES

warehouse_db-# (2,'15-APR-2008',1500,'Boston');

INSERT 0 0

warehouse_db=# INSERT INTO sales_record_listpart

warehouse_db-# (id, sales_date, sales_amount, city)

warehouse_db-# VALUES

warehouse_db-# (3,'16-APR-2008',1800,'Islamabad');

INSERT 0 0

warehouse_db=# INSERT INTO sales_record_listpart

warehouse_db-# (id, sales_date, sales_amount, city)

warehouse_db-# VALUES

warehouse_db-# (4,'20-APR-2008',1300,'new york');

INSERT 0 0

When you perform the preceding INSERT statements, you will observe that the INSERT query returns the INSERT 0 0 message; this is because the record is inserted in the child tables instead of the master tables.

Perform SELECT on salse_record_list1 to verify that the record is inserted as expected in the following manner:

warehouse_db=# SELECT * FROM sales_record_list1;

id | sales_date | sales_amount |   city

----+------------+--------------+----------

1 | 2008-04-15 |         1200 | sydney

4 | 2008-04-20 |         1300 | new york

(2 rows)

Perform SELECT on sales_record_list2 to verify that the record is inserted as expected in the following manner:

warehouse_db=# SELECT * FROM sales_record_list2;

id | sales_date | sales_amount |   city

----+------------+--------------+-----------

2 | 2008-04-15 |         1500 | Boston

3 | 2008-04-16 |         1800 | Islamabad

(2 rows)

Perform SELECT on sales_record_listpart to verify that the record is inserted as expected in the following manner:

warehouse_db=# SELECT * FROM sales_record_listpart;

id | sales_date | sales_amount |   city

----+------------+--------------+-----------

1 | 2008-04-15 |         1200 | sydney

4 | 2008-04-20 |         1300 | new york

2 | 2008-04-15 |         1500 | Boston

3 | 2008-04-16 |         1800 | Islamabad

(4 rows)