How to speed up insertion performance in PostgreSQL
- Disable any triggers on the table
- Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Change table to UNLOGGEDtable without indexes, then change it to logged and add the indexes.Unfortunately in PostgreSQL 9.4 there's no support for changing tables fromUNLOGGEDto logged. 9.5 addsALTER TABLE ... SET LOGGEDto permit you to do this.- Remove Foreign Key Constraints
- If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
- If possible, use
COPYinstead ofINSERTs - If you can't use
COPYconsider using multi-valuedINSERTs if practical. Don't try to list too many values in a singleVALUESthough; those values have to fit in memory a couple of times over, so keep it to a few hundred per statement. - Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
- Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
- Use
synchronous_commit=offand a hugecommit_delayto reduce fsync() costs. This won't help much if you've batched your work into big transactions, though. INSERTorCOPYin parallel from several connections. How many depends on your hardware's disk subsystem; as a rule of thumb, you want one connection per physical hard drive if using direct attached storage.- Set a high
checkpoint_segmentsvalue and enablelog_checkpoints. Look at the PostgreSQL logs and make sure it's not complaining about checkpoints occurring too frequently. - If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
fsync=off, start Pg, do your import, then (vitally) stop Pg and setfsync=onagain. See WAL configuration. Do not do this if there is already any data you care about in any database on your PostgreSQL install. If you setfsync=offyou can also setfull_page_writes=off; again, just remember to turn it back on after your import to prevent database corruption and data loss. See non-durable settings in the Pg manual. - Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql
https://www.postgresql.org/docs/9.4/static/populate.html
How to speed up insertion performance in PostgreSQL的更多相关文章
- PostgreSQL相关的软件,库,工具和资源集合
PostgreSQL相关的软件,库,工具和资源集合. 备份 wal-e - Simple Continuous Archiving for Postgres to S3, Azure, or Swif ...
- 【转载】利用window.performance.timing进行性能分析
利用window.performance.timing进行性能分析 性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备 ...
- 利用window.performance.timing进行性能分析
性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备加载页面的起始时间 // .unloadEventStart 如果前一个 ...
- 数据库新秀 postgresql vs mongo 性能PK
前几天看了一篇文章<High Performance JSON PostgreSQL vs. MongoDB> 发布在Percona Live Europe 2017 作者是<Dom ...
- Index-Only Scans and Covering Indexes
小结: 1.覆盖索引 回表 2. All indexes in PostgreSQL are secondary indexes, meaning that each index is stored ...
- Doherty Threshold
Prior to the publication of the IBM technical paper behind what commonly known today as the Doherty ...
- Android WIFI 分析(二)
本文介绍Wifi 分析线路二:在Setting中打开WiFi功能.扫描网络以及连接网络的流程. WifiSettings 无线网络设置界面 WifiEnabler 相当于无线网络设置开关 WifiDi ...
- sandy bridge
SANDY BRIDGE SPANS GENERATIONS Intel Focuses on Graphics, Multimedia in New Processor Design By Li ...
- 39. Volume Rendering Techniques
Milan Ikits University of Utah Joe Kniss University of Utah Aaron Lefohn University of California, D ...
随机推荐
- 初学DirectX(1)
初学Direct X (1) Direct3D设备用于访问视频卡的帧缓冲区,以及后台缓冲区.由于IDE是vs2013,默认安装了direct 9,只需要在使用头文件(1)并像使用库文件(2)即可 #i ...
- 前端开发工程师 - 01.页面制作 - 第4章.CSS
第4章.CSS CSS简介 Cascading Style Sheet 层叠样式表:定义页面中的表现样式 history: CSS1(1996)--CSS2(1998)--着手CSS3草案(拆分成很多 ...
- appium启动APP配置参数:
一.Android启动app python启动脚本如下: from appium import webdriver desired_caps = {} desired_caps['plat ...
- 操作系统及Python解释器工作原理讲解
操作系统介绍 操作系统位于计算机硬件与应用软件之间 是一个协调.管理.控制计算机硬件资源与软件资源的控制程序 操作系统功能: 控制硬件 把对硬件复杂的操作封装成优美简单的接口(文件),给用户或者应用程 ...
- 安装HIVE
参考:https://cwiki.apache.org/confluence/display/Hive/GettingStarted 1.下载hive安装包 到apache官网或者其它地方下载 ...
- Sum of Consecutive Prime Numbers(素数打表+尺取)
Description Some positive integers can be represented by a sum of one or more consecutive prime numb ...
- Sail
DescriptionThe polar bears are going fishing. They plan to sail from (sx,?sy) to (ex,?ey). However, ...
- 20172330 2017-2018-1 《Java程序设计》第九周学习总结
20172330 2017-2018-1 <程序设计与数据结构>第九周学习总结 教材学习内容总结 本周的学习包括两章内容,分别为异常和递归. 异常 错误和异常都是对象,代表非正常情况或者无 ...
- Sparsity Invariant CNNs
文章链接 Abstract 本文研究稀疏输入下的卷积神经网络,并将其应用于稀疏的激光扫描数据的深度信息完成实验.首先,我们表明,即使当丢失数据的位置提供给网络时,传统卷积网络在应用于稀疏数据时性能也很 ...
- C# .net 获取外网ip
public string GetIP() { string strUrl = "http://www.ip138.com/ip2city.asp"; //获得IP的网址了 Uri ...