【转载】Fast Inserts to PostgreSQL with JDBC and COPY FROM
source: http://rostislav-matl.blogspot.com/2011/08/fast-inserts-to-postgresql-with-jdbc.html
Thanks !
Fast Inserts to PostgreSQL with JDBC and COPY FROM
For the purpose of the test I created following table:
CREATE TABLE measurement
(
measurement_id bigint NOT NULL,
valid_ts timestamp with time zone NOT NULL,
measurement_value numeric(19,4) NOT NULL,
CONSTRAINT pk_mv_raw PRIMARY KEY (measurement_id, valid_ts)
)
WITH (OIDS=FALSE)
I decided to test the insertion of 1000 records to the table. The data for the recors was generated before running of any of test methods. Four test methods were created to reflect ususal approaches:
- VSI (Very Stupid Inserts) - executing queries made of concatenated Strings one by one
- SPI (Stupid Prepared Inserts) - similar to VSI but using prepared statements
- BPI (Batched Prepared Inserts) - prepared inserts, executed in batches of various length
- CPI (Copy Inserts) - inserts based on COPY FROM, executed in batches of various length
Prior to each inserts the table is cleared, the same after all data are succesfully inserted. Commit is called only once in each test method, following all the insert calls. The following code exerpts illustrate the above listed approaches:
VSI
for (int i=0; i<testSize; i++)
{
insertSQL = "insert into measurement values ("
+ measurementIds[i] +",'"+ timestamps[i] +"',"+values[i]+")";
insert.execute(insertSQL);
}
SPI
PreparedStatement insert = conn.prepareStatement("insert into measurement values (?,?,?)");
for (int i=0; i<testSize; i++)
{
insert.setLong(1,measurementIds[i]);
insert.setTimestamp(2, timestamps[i]);
insert.setBigDecimal(3, values[i]);
insert.execute();
}
BPI
PreparedStatement insert = conn.prepareStatement("insert into measurement values (?,?,?)");
for (int i=0; i<testSize; i++)
{
insert.setLong(1,measurementIds[i]);
insert.setTimestamp(2, timestamps[i]);
insert.setBigDecimal(3, values[i]);
insert.addBatch();
if (i % batchSize == 0) { insert.executeBatch(); }
}
insert.executeBatch();
CPI
StringBuilder sb = new StringBuilder();
CopyManager cpManager = ((PGConnection)conn).getCopyAPI();
PushbackReader reader = new PushbackReader( new StringReader(""), 10000 );
for (int i=0; i<testSize; i++)
{
sb.append(measurementIds[i]).append(",'")
.append(timestamps[i]).append("',")
.append(values[i]).append("\n");
if (i % batchSize == 0)
{
reader.unread( sb.toString().toCharArray() );
cpManager.copyIn("COPY measurement FROM STDIN WITH CSV", reader );
sb.delete(0,sb.length());
}
}
reader.unread( sb.toString().toCharArray() );
cpManager.copyIn("COPY measurement FROM STDIN WITH CSV", reader );
I hoped to get some improvements for using COPY FROM instead of batched inserts but not expected no big gain. But the results were a pleasant surprise. For a batch of size 50 (as defined in the original aplication I wanted to improve) the COPY FROM gave 40% improvement. I expect some improvements when data come from a stream and skip the StringBuffer-with-PushbackReader exercise.
See the graphs yourself - the number following the method abbreviation is the size of the batch.
![]() |
Average time in milliseconds |
![]() |
All the 200 runs individually |
【转载】Fast Inserts to PostgreSQL with JDBC and COPY FROM的更多相关文章
- 数据库语言(三):MySQL、PostgreSQL、JDBC
MySQL MySQL资料很多,这里只给出一个在论坛博客中最常用的操作:分页 mysql> select pname from product limit 10,20; limit的第一个参数是 ...
- (转载)SQL Server 2008 连接JDBC详细图文教程
点评:SQL Server 2008是目前windows上使用最多的sql数据库,2008的安装机制是基于framework重写的,特点是非常耗时间SQL Server 2008是目前windows上 ...
- postgresql Java JDBC 一次性传入多个参数到 in ( ?) - multple/list parameters
经常不清楚需要传入多少个参数到 IN () 里面,下面是简单方法: 方法 1 - in ( SELECT * FROM unnest(?)) ) Integer[] ids={1,2,3}; ...
- 【转载】C#的DataTable类Clone及Copy方法的区别
在C#中的Datatable类中,Clone方法和Copy方法都可以用来复制当前的DataTable对象,但DataTable类中的Clone方法和Copy方法还是有区别的,Clone方法只复制结构信 ...
- PostgreSQL数据导出导入COPY
[postgres@DELL-R720 bin]$ ./psql -p 6432psql (9.4.5)Type "help" for help. postgres=# postg ...
- 编写postgresql函数执行循环copy命令导入大数据
CREATE OR REPLACE FUNCTION copyData() RETURNS boolean AS $BODY$ DECLARE i int; begin i :=1; FOR i IN ...
- JMeter学习(八)JDBC测试计划-连接Oracle(转载)
转载自 http://www.cnblogs.com/yangxia-test 一.测试环境准备 Oracle:10g JDBC驱动:classes12.jar oracle安装目录下(orac ...
- 数据库jdbc链接:mysql, oracle, postgresql
#db mysql#jdbc.driver=com.mysql.jdbc.Driver#jdbc.url=jdbc:mysql://localhost:3306/mysql?&useUnico ...
- PostgreSQL相关的软件,库,工具和资源集合
PostgreSQL相关的软件,库,工具和资源集合. 备份 wal-e - Simple Continuous Archiving for Postgres to S3, Azure, or Swif ...
随机推荐
- substring,substr,和slice的区别详解。
1.Substring(x,y) : 输出一个字符串,当其中只有一个参数时,会输出从x开始到结尾的String. 举例: var str="hello"; conso ...
- sed详细分析
[一.简单描述] sed命令类似命令行的文本编辑器,以行为单位(见注1).除非带命令i(in-place)否则源文件内容并不会被更新. [二.使用] [2.1.使用方式] 存在两种使用方式: 1. ...
- php-数据库访问--数据修改
主页面元素修改脚本 <?php $code = $_GET["c"]; //造连接对象 $db = new MySQLi("localhost",&quo ...
- lucene教程简介
1 lucene简介 1.1 什么是lucene Lucene是一个全文搜索框架,而不是应用产品.因此它并不像www.baidu.com 或者google Desktop那么拿来就能用,它只是 ...
- 552 you must authentication
配置邮箱到outlook时 出现以下错误: 发送测试电子邮件消息: 无法发送此邮件.请在帐户属性中验证电子邮件地址. 响应服务器: 552 you must authentication 需要在”其 ...
- 台球游戏的核心算法和AI(1)
前言: 08年的时候, 写过一个台球游戏, 用的是java, 不过代码真的是用传说中的神器notepad写的(你信吗? 其实是用GVIM写的, ^_^), 很多类都在同一java文件中编写. 可见当时 ...
- java中的transient关键词
以下内容全部参考自:http://www.cnblogs.com/lanxuezaipiao/p/3369962.html,有些直接复制了. 1. transient的作用 实体类实现了Seriliz ...
- Qt之阴影边框(转)
原文地址:http://blog.sina.com.cn/s/blog_a6fb6cc90101eoc7.html 阴影边框很常见,诸如360以及其他很多软件都有类似效果,了解CSS3的同学们应该都知 ...
- lua的栈
lua的栈是从栈底到栈顶: lua_pushstring(L, "test1");lua_pushstring(L, "test2");lua_pushstri ...
- 知识积累:CGI,FastCGI,PHP-CGI与PHP-FPM
CGICGI全称是“公共网关接口”(Common Gateway Interface),HTTP服务器与你的或其它机器上的程序进行“交谈”的一种工具,其程序须运行在网络服务器上.CGI可以用任何一种语 ...