A blog on java performance and optimization. On JDBC, Hibernate, caching, algorithms, profiling and anything that can make your code run faster.

 
 
 
 
 
 
 

Sunday, June 21, 2015

JDBC fetch size - commonly forgotten setting

 
JDBC fetch size parameter can significantly reduce the result set fetching time. It can be set on any Statement or PreparedStatement object. Many frameworks like Spring or Hibernate give you very convenient API to do this.

Although this parameter is widely available, I discovered that a lot of projects I have visited did not configure the fetch size and thus was using its default value. Yet, the default value can result in a poor performance on some jdbc drivers.

I want to show you how much you can improve your fetching performance if you adjust the fetch size to your statements. The scale of improvement depends on the driver you use.

Oracle jdbc driver

Assume we have table with 1 million rows and for any reason we have to fetch all records into JVM. How fast you can get all data? What fetch size will Oracle use if you don’t set it explicitly?

Figure 1 - fetching 1M rows with different fetchSize values (local oracle database)

Figure 1 shows fetching times for different fetch size values for Oracle database. In this example database and java application are located on the same machine. I will show later how it looks for a remote database.

Setting Fetch Size with standard JDBC calls

This is how you can set fetch size for given PreparedStatement using JDBC API:

  PreparedStatement stmt = null;
  ResultSet rs = null;
   
  try {
  stmt = conn. prepareStatement("select a, b, c from table");
  stmt.setFetchSize(200);
   
  rs = stmt.executeQuery();
  while (rs.next()) {
  ...
  }
  }

Lets see what happens if fetchSize property is set to 10. When rs.next() is called for first time, the oracle driver fetches first 10 records from database and store them in a memory buffer. So, for next 9 calls to rs.next() records are retrieved from this buffer. After the buffer is fully read, subsequent rs.next()will force driver to fetch a new bunch of rows (10) into the buffer.

So if we want to read 10k rows with fetch size set to 10, the driver will make 1000 round trips to the database using the underlying connection. If we set the fetchSize to 500 the driver will perform only 20 round trips to our database.

Look at Figure 1. Setting fetchSize to 100 gives you a 6 times shorter fetching time then with setting fetchSize to 10. Now, you should know that the default fetchSize for the oracle driver is 10...

Two important comments:

  • fetchSize can be set on each Statement or PreparedStatement or even on ResultSet. By default, ResultSet uses fetchSize of Statement from which is born. The default value for Statement or PreparedStatementis jdbc driver specific
  • fetchSize is only a hint for the driver – the Oracle driver respects this setting, while other drivers may ignore it and fetch all the records at once, for instance.
 

Setting Fetch Size with Spring JdbcTemplate

When using Spring jdbc support you can do this in 2 ways:

Ad hoc JdbcTemplate instance

  JdbcTemplate jdbc = new JdbcTemplate(dataSource);
  jdbc.setFetchSize(200);
   
  jdbc.query("select a, b, c from table",
   
  new RowCallbackHandler() {
  @Override
  public void processRow(ResultSet rs) throws SQLException {
  ...
  }
  }
  );

Shared JdbcTemplate instance

  public class MyJdbcDaoImpl extends JdbcDaoSupport implements MyJdbcDao {
   
  @Override
  protected void initTemplateConfig() {
  getJdbcTemplate().setFetchSize(200);
  }
   
  public MyResult loadAll() {
  final MyResult result = new MyResult();
   
  getJdbcTemplate().query("select a, b, c from table",
   
  new RowCallbackHandler() {
  @Override
  public void processRow(ResultSet rs) throws SQLException {
  ...
  result.add(...);
  }
  }
  );
   
  } // end of loadAll
  }

When implementing a DAO that extends JdbcDaoSupport every call to getJdbcTemplate() returns the same shared JdbcTemplate instance. You can mix this with ad-hoc instances. For example, override initTemplateConfig()to set the default for this DAO but use ad-hoc JdbcTemplate for selected queries.

JDBC fetch size的更多相关文章

  1. SQL Fetch size

    JDBC performance tuning with optimal fetch size February 1, 2009 31 Comments Tuning performance usin ...

  2. hibernate.properties官方属性用例(可用于hibernate.cfg.xml属性参考)

    ######################### Query Language ######################### ## define query language constant ...

  3. hibernate.cfg.xml文件的配置模板和不同数据库的配置參数

    (1)hibernate.cfg.xml文件的配置模板 <?xml version="1.0" encoding="UTF-8"?> <!DO ...

  4. Hibernate配置文件中配置各种数据库的driver、URL

    hibernate.properties ######################### Query Language ######################### ## define qu ...

  5. Hibernate配置文件中配置各种数据库链接

    hibernate.properties ###################### ### Query Language ### ###################### ## define ...

  6. Spark SQL笔记——技术点汇总

    目录 概述 原理 组成 执行流程 性能 API 应用程序模板 通用读写方法 RDD转为DataFrame Parquet文件数据源 JSON文件数据源 Hive数据源 数据库JDBC数据源 DataF ...

  7. hibernate学习(一)配置,导包

    框架的作用 学过javaWeb基础的已经对web层 jsp  servlet   ,service  层  ,dao层的jdbc .DBUtils 有了很深的了解 并编写代码实现某种功能 为了提高开发 ...

  8. JAVA PERSISTENCE API (JPA)

    13.2.1. About JPA The Java Persistence API (JPA) is the standard for using persistence in Java proje ...

  9. java框架之Hibernate(1)-简介及初使用

    简介 hibernate 是一个开源 ORM ( Object / Relationship Mipping ) 框架,它是对象关联关系映射的持久层框架,它对 JDBC 做了轻量级的封装,而我们 ja ...

随机推荐

  1. Android BroadcastReceiver解析

    目录   示意图 1. 定义 即 广播,是一个全局的监听器,属于Android四大组件之一 Android 广播分为两个角色:广播发送者.广播接收者 2. 作用 监听 / 接收 应用 App 发出的广 ...

  2. Mask RCNN 学习笔记

    下面会介绍基于ResNet50的Mask RCNN网络,其中会涉及到RPN.FPN.ROIAlign以及分类.回归使用的损失函数等 介绍时所采用的MaskRCNN源码(python版本)来源于GitH ...

  3. oracle新建对象 权限管理

    代码 CREATE USER target IDENTIFIED BY target ; GRANT CONNECT, RESOURCE TO target; 刚刚创建的oracle实例中会内建两个用 ...

  4. python正则表达式二[转]

    原文:http://www.cnblogs.com/huxi/archive/2010/07/04/1771073.html 1. 正则表达式基础 1.1. 简单介绍 正则表达式并不是Python的一 ...

  5. Struts S2-052漏洞利用

    昨天在FreeBuf上看到[9月6日更新]漏洞预警 | 高危Struts REST插件远程代码执行漏洞(S2-052) 然而一直复现不了,今天又试了下竟然成功了. 由于水表查的较严,就不冒险搞别人的服 ...

  6. Apriori 算法python实现

    1. Apriori算法简介 Apriori算法是挖掘布尔关联规则频繁项集的算法.Apriori算法利用频繁项集性质的先验知识,通过逐层搜索的迭代方法,即将K-项集用于探察(k+1)项集,来穷尽数据集 ...

  7. 命令行command line 使用 http proxy的设置方法 Setting Up HTTP Proxy in Terminal

    Step 1: Install Shadowsocks Client Shadowsocks is an open-source proxy project to help people visit ...

  8. js里的回调函数

    function a(callback)  // 定义一个函数 ,需要传入的参数是callback  然后callback的类型为一个函数{console.log("callback还表示传 ...

  9. Qt Excel

    在pro文件添加 QT +=axcontainer 头文件 #include <QAxObject> void MainWindow::on_btnSelectFileDialog_cli ...

  10. PYTHON-模块定义 搜索路径

    模块是什么: ***** 模块 是一系列功能的集合体 一个py文件就是一个模块 一个函数就是一个功能 例如 A.py 文件名A.py 模块名 A 模块有哪些来源 内置 第三方 自定义 模块有四种通用的 ...