How to speed up insertion performance in PostgreSQL
- Disable any triggers on the table
- Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Change table to UNLOGGEDtable without indexes, then change it to logged and add the indexes.Unfortunately in PostgreSQL 9.4 there's no support for changing tables fromUNLOGGEDto logged. 9.5 addsALTER TABLE ... SET LOGGEDto permit you to do this.- Remove Foreign Key Constraints
- If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
- If possible, use
COPYinstead ofINSERTs - If you can't use
COPYconsider using multi-valuedINSERTs if practical. Don't try to list too many values in a singleVALUESthough; those values have to fit in memory a couple of times over, so keep it to a few hundred per statement. - Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
- Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
- Use
synchronous_commit=offand a hugecommit_delayto reduce fsync() costs. This won't help much if you've batched your work into big transactions, though. INSERTorCOPYin parallel from several connections. How many depends on your hardware's disk subsystem; as a rule of thumb, you want one connection per physical hard drive if using direct attached storage.- Set a high
checkpoint_segmentsvalue and enablelog_checkpoints. Look at the PostgreSQL logs and make sure it's not complaining about checkpoints occurring too frequently. - If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
fsync=off, start Pg, do your import, then (vitally) stop Pg and setfsync=onagain. See WAL configuration. Do not do this if there is already any data you care about in any database on your PostgreSQL install. If you setfsync=offyou can also setfull_page_writes=off; again, just remember to turn it back on after your import to prevent database corruption and data loss. See non-durable settings in the Pg manual. - Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql
https://www.postgresql.org/docs/9.4/static/populate.html
How to speed up insertion performance in PostgreSQL的更多相关文章
- PostgreSQL相关的软件,库,工具和资源集合
PostgreSQL相关的软件,库,工具和资源集合. 备份 wal-e - Simple Continuous Archiving for Postgres to S3, Azure, or Swif ...
- 【转载】利用window.performance.timing进行性能分析
利用window.performance.timing进行性能分析 性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备 ...
- 利用window.performance.timing进行性能分析
性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备加载页面的起始时间 // .unloadEventStart 如果前一个 ...
- 数据库新秀 postgresql vs mongo 性能PK
前几天看了一篇文章<High Performance JSON PostgreSQL vs. MongoDB> 发布在Percona Live Europe 2017 作者是<Dom ...
- Index-Only Scans and Covering Indexes
小结: 1.覆盖索引 回表 2. All indexes in PostgreSQL are secondary indexes, meaning that each index is stored ...
- Doherty Threshold
Prior to the publication of the IBM technical paper behind what commonly known today as the Doherty ...
- Android WIFI 分析(二)
本文介绍Wifi 分析线路二:在Setting中打开WiFi功能.扫描网络以及连接网络的流程. WifiSettings 无线网络设置界面 WifiEnabler 相当于无线网络设置开关 WifiDi ...
- sandy bridge
SANDY BRIDGE SPANS GENERATIONS Intel Focuses on Graphics, Multimedia in New Processor Design By Li ...
- 39. Volume Rendering Techniques
Milan Ikits University of Utah Joe Kniss University of Utah Aaron Lefohn University of California, D ...
随机推荐
- Java开发工程师(Web方向) - 03.数据库开发 - 第1章.JDBC
第1章--JDBC JDBC基础 通过Java Database Connectivity可以实现Java程序对后端数据库的访问 一个完整的数据库部署架构,通常是由客户端和服务器端两部分组成 客户端封 ...
- 关于javascript的一个小问题,请问有人看出啥问题吗?
最近学习javascript,有一个问题挺奇怪的,先贴出代码: function binarySearch(){ var arr = [0,1,2,3]; var res = actbinarySea ...
- Android开发-API指南-<uses-permission>
<uses-permission> 英文原文:http://developer.android.com/guide/topics/manifest/uses-permission-elem ...
- python常用命令—ipython3环境下获取某个文件夹下的文件列表
import os os.listdir('文件夹路径')
- LeetCode 169. Majority Element - majority vote algorithm (Java)
1. 题目描述Description Link: https://leetcode.com/problems/majority-element/description/ Given an array ...
- jquery中的$(document).ready()、JavaScript中的window.onload()以及body中的onload()、DomContentLoaded()区别
$().ready().$(handler).$(document).ready(handler)均不是原生JS中的,都是jQuery中封装的方法.这些事件在当页面的dom节点加载完毕后就执行,无需等 ...
- c# 有无符号值进一步了解
1.编写过程中用到了short类型(有符号型,值范围含负值).两个正数之和得负. 改为int或unsigned short 均可. 2.注意,short型(-32768,32767)举例:做自加运算, ...
- poj 3009 (深搜求最短路)
题目大意就是求在特定规则下的最短路,这个规则包含了消除障碍的操作.用BFS感觉选择消除障碍的时候不同路径会有影响,用DFS比较方便状态的还原(虽然效率比较低),因此这道题目采用DFS来写. 写的第一次 ...
- Java微笔记(1)
一,Arrays 类是 Java 中提供的一个工具类,在 java.util 包中.该类中包含了一些方法用来直接操作数组,比如可直接实现数组的排序.搜索 1. 排序 语法: Arrays.sort(数 ...
- DAY6敏捷冲刺
站立式会议 工作安排 (1)服务器配置 服务器端项目结构调整 (2)数据库配置 单词学习记录+用户信息 (3)客户端 客户端项目结构调整,代码功能分离 燃尽图 燃尽图有误,已重新修改,先贴卡片的界面, ...