How to speed up insertion performance in PostgreSQL
- Disable any triggers on the table
- Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Change table to UNLOGGEDtable without indexes, then change it to logged and add the indexes.Unfortunately in PostgreSQL 9.4 there's no support for changing tables fromUNLOGGEDto logged. 9.5 addsALTER TABLE ... SET LOGGEDto permit you to do this.- Remove Foreign Key Constraints
- If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
- If possible, use
COPYinstead ofINSERTs - If you can't use
COPYconsider using multi-valuedINSERTs if practical. Don't try to list too many values in a singleVALUESthough; those values have to fit in memory a couple of times over, so keep it to a few hundred per statement. - Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
- Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
- Use
synchronous_commit=offand a hugecommit_delayto reduce fsync() costs. This won't help much if you've batched your work into big transactions, though. INSERTorCOPYin parallel from several connections. How many depends on your hardware's disk subsystem; as a rule of thumb, you want one connection per physical hard drive if using direct attached storage.- Set a high
checkpoint_segmentsvalue and enablelog_checkpoints. Look at the PostgreSQL logs and make sure it's not complaining about checkpoints occurring too frequently. - If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
fsync=off, start Pg, do your import, then (vitally) stop Pg and setfsync=onagain. See WAL configuration. Do not do this if there is already any data you care about in any database on your PostgreSQL install. If you setfsync=offyou can also setfull_page_writes=off; again, just remember to turn it back on after your import to prevent database corruption and data loss. See non-durable settings in the Pg manual. - Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql
https://www.postgresql.org/docs/9.4/static/populate.html
How to speed up insertion performance in PostgreSQL的更多相关文章
- PostgreSQL相关的软件,库,工具和资源集合
PostgreSQL相关的软件,库,工具和资源集合. 备份 wal-e - Simple Continuous Archiving for Postgres to S3, Azure, or Swif ...
- 【转载】利用window.performance.timing进行性能分析
利用window.performance.timing进行性能分析 性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备 ...
- 利用window.performance.timing进行性能分析
性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备加载页面的起始时间 // .unloadEventStart 如果前一个 ...
- 数据库新秀 postgresql vs mongo 性能PK
前几天看了一篇文章<High Performance JSON PostgreSQL vs. MongoDB> 发布在Percona Live Europe 2017 作者是<Dom ...
- Index-Only Scans and Covering Indexes
小结: 1.覆盖索引 回表 2. All indexes in PostgreSQL are secondary indexes, meaning that each index is stored ...
- Doherty Threshold
Prior to the publication of the IBM technical paper behind what commonly known today as the Doherty ...
- Android WIFI 分析(二)
本文介绍Wifi 分析线路二:在Setting中打开WiFi功能.扫描网络以及连接网络的流程. WifiSettings 无线网络设置界面 WifiEnabler 相当于无线网络设置开关 WifiDi ...
- sandy bridge
SANDY BRIDGE SPANS GENERATIONS Intel Focuses on Graphics, Multimedia in New Processor Design By Li ...
- 39. Volume Rendering Techniques
Milan Ikits University of Utah Joe Kniss University of Utah Aaron Lefohn University of California, D ...
随机推荐
- 利用爬虫、SMTP和树莓派3B发送邮件(爬取墨迹天气预报信息)
-----------------------------------------学无止境----------------------------------------- 前言:大家好,欢迎来到誉雪 ...
- 孤荷凌寒自学python第七十八天开始写Python的第一个爬虫8
孤荷凌寒自学python第七十八天开始写Python的第一个爬虫8 (完整学习过程屏幕记录视频地址在文末) 今天在上一天的基础上继续完成对我的第一个代码程序的书写. 到今天止基本完成了对docx模块针 ...
- python3-声音处理
先来说下二进制读写文件,这需要struct库 #二进制文件读写 import struct a= b=- # print(struct.pack("h",b)) # print(s ...
- leetcode个人题解——#18 4sums
在3sums的基础上加了一层循环. class Solution { public: vector<vector<int>> fourSum(vector<int> ...
- EasyJSWebView原理分析
概述 在iOS6之前,native只能调用webiew里的js代码,官方没有提供js调用native方法的接口.到了iOS7,官方提供了JSContext用来与js交互,native和js可以双向调用 ...
- 【IdentityServer4文档】- 支持和咨询选项
支持和咨询选项 我们为 IdentityServer 提供多个免费和商业支持及咨询选项. 免费支持 免费支持是基于社区的,而且使用的是公共论坛 StackOverflow 有越来越多的使用 Ident ...
- .net下将exe改造为既能双击独立运行又能注册为windows服务
最近项目中需要将一些业务的处理程序改造为windows服务,但是考虑到实际需求,也需要能够直接双击运行这些处理程序.首先第一步想到的就是原来的项目不变,只需要在加一个windows服务的项目就行.但是 ...
- DELPHI dbgrid 选中的是第几行 怎么判断?
使用DataSource.DataSet.RecNo可以得到dbgrid选中的是第几行,示例代码如下: procedure TForm1.btn1Click(Sender: TObject); beg ...
- window.navigator.standalone 检测iOS WebApp是否运行在全屏模式
iOS上的Safari浏览器可以让Web应用程序全屏显示,以取得类似本地应用的显示效果.但是这需要用户把Web应用程序的图标添加到主屏幕才可以.作为开发者,为了更好的显示效果,我们可能希望自己开发的W ...
- byte字节的输入流 建议使用字节数组形式接受内容 因为当输入流是汉字时候 会超出字节的范围 出现无法读取的现象
byte字节的输入流 建议使用字节数组形式接受内容 因为当输入流是汉字时候 会超出字节的范围 出现无法读取的现象