postgresql大批量数据导入方法
一直没有好好关注这个功能,昨天看了一下,数据库插入有瓶颈,今天研究了一下:
主要有以下方案:
1.使用copy从文件导入:
copy table_001(a, b, "f", d, c, "e") from 'd:/data1.txt' (delimiter ',');
速度极快:
不带索引:
查询成功: 共计 69971 行受到影响,耗时: 4351 毫秒(ms)。
查询成功: 共计 69971 行受到影响,耗时: 4971 毫秒(ms)。
带索引:
查询成功: 共计 69971 行受到影响,耗时: 15582 毫秒(ms)。
查询成功: 共计 69971 行受到影响,耗时: 12833 毫秒(ms)。
需要做的就是定时生成临时数据文件,并不断的切换,清除。
2. 使用multi-insert格式的sql
类似: insert into test values('asd', 'adewf', 12),('asd2', 'adewf2', 12);
目前采用此方案,改动不大,只是修改了一下 sql 的格式,目前满足要求(大约25万条记录每分钟,合4200每秒),所以暂时采用它。
3. 关闭自动提交,使用insert或者multi-insert格式sql,插入大量数据
目前未测试,不过此方案效果具网上介绍应该也不错的。
4. 采用临时表
这个方案备选,临时表为了加快速度,应该不加任何索引与日志,数据稳定后再加索引与限制,压缩数据,进行vacuum 等数据优化,这需要与分表结合使用比较好。
5. 调整数据库参数,这个是提高数据库整体性能的
网上介绍这几个优化参数:shared_buffers、work_mem、effective_cache_size、maintence_work_mem
这些可以配置起来使用,详细请参考 postgresql-9.2-A4.pdf 中的 Chapter 14. Performance Tips。
One might need to insert a large amount of data when first populating a database. This section contains
some suggestions on how to make this process as efficient as possible.
14.4.1. Disable Autocommit
When using multiple INSERTs, turn off autocommit and just do one commit at the end. (In plain
SQL, this means issuing BEGIN at the start and COMMIT at the end. Some client libraries might do this
behind your back, in which case you need to make sure the library does it when you want it done.) If
you allow each insertion to be committed separately, PostgreSQL is doing a lot of work for each row
that is added. An additional benefit of doing all insertions in one transaction is that if the insertion of
one row were to fail then the insertion of all rows inserted up to that point would be rolled back, so
you won’t be stuck with partially loaded data.
14.4.2. Use COPY
Use COPY to load all the rows in one command, instead of using a series of INSERT commands. The
COPY command is optimized for loading large numbers of rows; it is less flexible than INSERT, but
incurs significantly less overhead for large data loads. Since COPY is a single command, there is no
need to disable autocommit if you use this method to populate a table.
If you cannot use COPY, it might help to use PREPARE to create a prepared INSERT statement, and
then use EXECUTE as many times as required. This avoids some of the overhead of repeatedly parsing
and planning INSERT. Different interfaces provide this facility in different ways; look for “prepared
statements” in the interface documentation.
Note that loading a large number of rows using COPY is almost always faster than using INSERT, even
if PREPARE is used and multiple insertions are batched into a single transaction.
COPY is fastest when used within the same transaction as an earlier CREATE TABLE or TRUNCATE
command. In such cases no WAL needs to be written, because in case of an error, the files containing the newly loaded data will be removed anyway. However, this consideration only applies when
wal_level is minimal as all commands must write WAL otherwise.
367
14.4.3. Remove Indexes
If you are loading a freshly created table, the fastest method is to create the table, bulk load the table’s
data using COPY, then create any indexes needed for the table. Creating an index on pre-existing data
is quicker than updating it incrementally as each row is loaded.
If you are adding large amounts of data to an existing table, it might be a win to drop the indexes,
load the table, and then recreate the indexes. Of course, the database performance for other users
might suffer during the time the indexes are missing. One should also think twice before dropping a
unique index, since the error checking afforded by the unique constraint will be lost while the index
is missing.
14.4.4. Remove Foreign Key Constraints
Just as with indexes, a foreign key constraint can be checked “in bulk” more efficiently than row-byrow. So it might be useful to drop foreign key constraints, load data, and re-create the constraints.
Again, there is a trade-off between data load speed and loss of error checking while the constraint is
missing.
What’s more, when you load data into a table with existing foreign key constraints, each new row
requires an entry in the server’s list of pending trigger events (since it is the firing of a trigger that
checks the row’s foreign key constraint). Loading many millions of rows can cause the trigger event
queue to overflow available memory, leading to intolerable swapping or even outright failure of the
command. Therefore it may be necessary, not just desirable, to drop and re-apply foreign keys when
loading large amounts of data. If temporarily removing the constraint isn’t acceptable, the only other
recourse may be to split up the load operation into smaller transactions.
14.4.5. Increase maintenance_work_mem
Temporarily increasing the maintenance_work_mem configuration variable when loading large
amounts of data can lead to improved performance. This will help to speed up CREATE INDEX
commands and ALTER TABLE ADD FOREIGN KEY commands. It won’t do much for COPY itself, so
this advice is only useful when you are using one or both of the above techniques.
14.4.6. Increase checkpoint_segments
Temporarily increasing the checkpoint_segments configuration variable can also make large data
loads faster. This is because loading a large amount of data into PostgreSQL will cause checkpoints
to occur more often than the normal checkpoint frequency (specified by the checkpoint_timeout
configuration variable). Whenever a checkpoint occurs, all dirty pages must be flushed to disk. By
increasing checkpoint_segments temporarily during bulk data loads, the number of checkpoints
that are required can be reduced.
14.4.7. Disable WAL Archival and Streaming Replication
When loading large amounts of data into an installation that uses WAL archiving or streaming replication, it might be faster to take a new base backup after the load has completed than to process
a large amount of incremental WAL data. To prevent incremental WAL logging while loading, disable archiving and streaming replication, by setting wal_level to minimal, archive_mode to off, and
max_wal_senders to zero. But note that changing these settings requires a server restart.
Aside from avoiding the time for the archiver or WAL sender to process the WAL data, doing this
will actually make certain commands faster, because they are designed not to write WAL at all if
wal_level is minimal. (They can guarantee crash safety more cheaply by doing an fsync at the
end than by writing WAL.) This applies to the following commands:
• CREATE TABLE AS SELECT
• CREATE INDEX (and variants such as ALTER TABLE ADD PRIMARY KEY)
• ALTER TABLE SET TABLESPACE
• CLUSTER
• COPY FROM, when the target table has been created or truncated earlier in the same transaction
14.4.8. Run ANALYZE Afterwards
Whenever you have significantly altered the distribution of data within a table, running ANALYZE
is strongly recommended. This includes bulk loading large amounts of data into the table. Running
ANALYZE (or VACUUM ANALYZE) ensures that the planner has up-to-date statistics about the table.
With no statistics or obsolete statistics, the planner might make poor decisions during query planning,
leading to poor performance on any tables with inaccurate or nonexistent statistics. Note that if the
autovacuum daemon is enabled, it might run ANALYZE automatically; see Section 23.1.3 and Section
23.1.6 for more information.
postgresql大批量数据导入方法的更多相关文章
- Java实现大批量数据导入导出(100W以上) -(二)导出
使用POI或JXLS导出大数据量(百万级)Excel报表常常面临两个问题: 1. 服务器内存溢出: 2. 一次从数据库查询出这么大数据,查询缓慢. 当然也可以分页查询出数据,分别生成多个Excel打包 ...
- Java实现大批量数据导入导出(100W以上) -(三)超过25列Excel导出
前面一篇文章介绍大数据量导出实现: Java实现大批量数据导入导出(100W以上) -(二)导出 这篇文章在Excel列较少时,按以上实际验证能很快实现生成.但如果列较多时用StringTemplat ...
- Java实现大批量数据导入导出(100W以上) -(一)导入
最近业务方有一个需求,需要一次导入超过100万数据到系统数据库.可能大家首先会想,这么大的数据,干嘛通过程序去实现导入,为什么不直接通过SQL导入到数据库. 大数据量报表导出请参考:Java实现大批量 ...
- [PHP]PHPOffice/PHPExcel数据导入方法
------------------------------------------------------------------------------------ /** * PHPExcel数 ...
- java大批量数据导入(MySQL)
© 版权声明:本文为博主原创文章,转载请注明出处 最近同事碰到大批量数据导入问题,因此也关注了一下.大批量数据导入主要存在两点问题:内存溢出和导入速率慢. 内存溢出:将文件中的数据全部取出放在集合中, ...
- hive数据导入方法
可以通过多种方式将数据导入hive表 1.通过外部表导入 用户在hive上建external表,建表的同时指定hdfs路径,在数据拷贝到指定hdfs路径的同时,也同时完成数据插入external表. ...
- odoo11 外部数据导入方法2
前面有一篇文章分析了如何使用2个分开的文件分别将外部数据导入到odoo对应的系统当中,如之前所说,是存在缺点的,现在测试将所有数据放入一个文件中将主表与从表的数据一次性导入,这样可以很方便的利用odo ...
- oracle中生成大批量数据的方法-下
方法五:使用PLSQL的数据生成器 首先测试环境建立:dept表 CREATE TABLE dept(deptno NUMBER(6),dname VARCHAR2(20),loc VARCHAR2( ...
- SqlBulkCopy实现大批量数据导入
//自增列重新生成:SqlBulkCopy bc = new SqlBulkCopy(conn) //自增列保留原值:SqlBulkCopy bc = new SqlBulkCopy(conn,Sql ...
随机推荐
- SEO网站优化方案
学习许多前辈的经验,看到一些比较有价值的seo优化方案,特记录一下,对照自己的操作之路,新人也可借鉴一二,下面是从卢松松博客看到的文章.高手直接跳过,请勿喷水. 一个完整的SEO优化方案主要由四个小组 ...
- linux gcc loudong
五事九思 (大连Linux主机维护) 大连linux维护qq群:287800525 首页 日志 相册 音乐 收藏 博友 关于我 日志 spcark_0.0.3_i386.src.t ...
- 【面试题】Round A China New Grad Test 2014总结
我也有够懒的,今天才跑来写总结,自觉面壁中… 上一篇是Practice Round,今天是Round A,五道题. 每次做完都想说,其实题不难..但在做的过程中总是会各种卡,只有自己一行一行实现了,才 ...
- (转)Learning to Rank for IR的评价指标—MAP,NDCG,MRR
转自:http://www.cnblogs.com/eyeszjwang/articles/2368087.html MAP(Mean Average Precision):单个主题的平均准确率是每篇 ...
- POJ 1961 2406 (KMP,最小循环节,循环周期)
关于KMP的最短循环节.循环周期,请戳: http://www.cnblogs.com/chenxiwenruo/p/3546457.html (KMP模板,最小循环节) POJ 2406 Powe ...
- 二维码(支持arc,苹果自带扫描,zbar扫描,二维码生成)
下载地址:http://pan.baidu.com/s/1mgvJzyo
- C# 6 —— 属性
记录一下 C# 6 有关属性的语法糖实现,C# 6 涉及到属性的新特性主要有 2 个:自动属性初始化.只读属性赋值与读取. 自动属性初始化(Auto-property initializers) C# ...
- VS2003 下GridControl的列显示成图片+文字的形式实现
public RC_CustomerSolicitListUC() { // 该调用是 Windows.Forms 窗体设计器所必需的. InitializeComponent(); // TODO: ...
- Good Bye 2014 D. New Year Santa Network 图论+期望
D. New Year Santa Network New Year is coming in Tree World! In this world, as the name implies, th ...
- C#程序重启自己
Process.Start(System.Reflection.Assembly.GetExecutingAssembly().Location); Application.Exit();