How to speed up insertion performance in PostgreSQL
- Disable any triggers on the table
- Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Change table to UNLOGGEDtable without indexes, then change it to logged and add the indexes.Unfortunately in PostgreSQL 9.4 there's no support for changing tables fromUNLOGGEDto logged. 9.5 addsALTER TABLE ... SET LOGGEDto permit you to do this.- Remove Foreign Key Constraints
- If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
- If possible, use
COPYinstead ofINSERTs - If you can't use
COPYconsider using multi-valuedINSERTs if practical. Don't try to list too many values in a singleVALUESthough; those values have to fit in memory a couple of times over, so keep it to a few hundred per statement. - Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
- Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
- Use
synchronous_commit=offand a hugecommit_delayto reduce fsync() costs. This won't help much if you've batched your work into big transactions, though. INSERTorCOPYin parallel from several connections. How many depends on your hardware's disk subsystem; as a rule of thumb, you want one connection per physical hard drive if using direct attached storage.- Set a high
checkpoint_segmentsvalue and enablelog_checkpoints. Look at the PostgreSQL logs and make sure it's not complaining about checkpoints occurring too frequently. - If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
fsync=off, start Pg, do your import, then (vitally) stop Pg and setfsync=onagain. See WAL configuration. Do not do this if there is already any data you care about in any database on your PostgreSQL install. If you setfsync=offyou can also setfull_page_writes=off; again, just remember to turn it back on after your import to prevent database corruption and data loss. See non-durable settings in the Pg manual. - Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql
https://www.postgresql.org/docs/9.4/static/populate.html
How to speed up insertion performance in PostgreSQL的更多相关文章
- PostgreSQL相关的软件,库,工具和资源集合
PostgreSQL相关的软件,库,工具和资源集合. 备份 wal-e - Simple Continuous Archiving for Postgres to S3, Azure, or Swif ...
- 【转载】利用window.performance.timing进行性能分析
利用window.performance.timing进行性能分析 性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备 ...
- 利用window.performance.timing进行性能分析
性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备加载页面的起始时间 // .unloadEventStart 如果前一个 ...
- 数据库新秀 postgresql vs mongo 性能PK
前几天看了一篇文章<High Performance JSON PostgreSQL vs. MongoDB> 发布在Percona Live Europe 2017 作者是<Dom ...
- Index-Only Scans and Covering Indexes
小结: 1.覆盖索引 回表 2. All indexes in PostgreSQL are secondary indexes, meaning that each index is stored ...
- Doherty Threshold
Prior to the publication of the IBM technical paper behind what commonly known today as the Doherty ...
- Android WIFI 分析(二)
本文介绍Wifi 分析线路二:在Setting中打开WiFi功能.扫描网络以及连接网络的流程. WifiSettings 无线网络设置界面 WifiEnabler 相当于无线网络设置开关 WifiDi ...
- sandy bridge
SANDY BRIDGE SPANS GENERATIONS Intel Focuses on Graphics, Multimedia in New Processor Design By Li ...
- 39. Volume Rendering Techniques
Milan Ikits University of Utah Joe Kniss University of Utah Aaron Lefohn University of California, D ...
随机推荐
- SecureCRT 注册
http://download.csdn.net/download/xia2011214228/9952983 1.下载后解压到安装目录 2.输入自己要注册的:name company 后genera ...
- Python全栈 进阶(进阶内容都在这了)
原文地址 https://yq.aliyun.com/articles/632754?spm=a2c4e.11155435.0.0.23eb3312feB6dG ................... ...
- MyBatis 注解配置及动态SQL
一.注解配置 目前MyBatis支持注解配置,用注解方式来替代映射文件,但是注解配置还是有点不完善,在开发中使用比较少,大部分的企业还是在用映射文件来进行配置.不完善的地方体现在于当数据表中的字段 ...
- 持续集成之TeamCity 配置
xcopy /S /Y CodeFirstDemo\CodefirstDemo.Web D:\publish\welcome\Web
- node.js应用--转载
最近,在向大学生们介绍 HTML5 的时候,我想要对他们进行问卷调查,并向他们显示实时更新的投票结果.鉴于此目的,我决定快速构建一个用于此目的的问卷调查应用程序.我想要一个简单的架构,不需要太多不同的 ...
- 2.hbase原理(未完待续)
hbase简介相关概念hmsterhregionserver表regionhstorememstorestorefilehfileblockcacheWALminorcompactmajorcompa ...
- Ubuntu使用时遇到的问题
启动时显示System program problem detected 解决办法: 打开命令行窗口:Ctrl+Alt+T 执行命令:sudo gedit /etc/default/apport 把e ...
- lintcode-6-合并排序数组
合并排序数组 合并两个排序的整数数组A和B变成一个新的数组. 样例 给出A=[1,2,3,4],B=[2,4,5,6],返回 [1,2,2,3,4,4,5,6] 挑战 你能否优化你的算法,如果其中一个 ...
- TCP系列14—重传—4、Karn算法和TSOPT的RTTM
一.Karn算法 在RTT采样测量过程中,如果一个数据包初传后,RTO超时重传,接着收到这个数据包的ACK报文,那么这个ACK报文是对应初传TCP报文还是对应重传TCP报文呢?这个问题就是retran ...
- vc6.0批量加注释
MATLAB批量加注释的方法非常简单明了,加注释是ctrl+R,去注释是ctrl+T 然后在VC中我对一条一条加注释的方法非常烦恼,我想也许会有简单的方法可以批量家注释.果然,先贴代码 '------ ...