使用solr的DIHandler 构建mysql大表全量索引,内存溢出问题的解决方法
solr官方给出的解决方式是:
DataImportHandler is designed to stream row one-by-one. It passes a fetch size value (default: 500) to Statement#setFetchSize which some drivers do not honor. For MySQL, add batchSize property to dataSource configuration with value -1. This will pass Integer.MIN_VALUE to the driver as the fetch size and keep it from going out of memory for large tables. Should look like: <dataSource type="JdbcDataSource" name="ds-2" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:8889/mysqldatabase" batchSize="-1" user="root" password="root"/>
说明:DataImportHandler 设计是支持按行获取的。它通过Statement#setFetchSize来设置每次获取的数量,默认是500个。然而一些驱动不支持设置fetchSize。对mysql来说,传递fetchSize属性值-1到Datasource配置中。它将将Integer.MIN_VALUE(-231,-2147483648 [0x80000000])传给驱动作为fetchsize,此时确保大表不会造成大表移除。
mysql官方给出的解释是:
ResultSet By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. If you are working with ResultSets that have a large number of rows or large values and cannot allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time. To enable this functionality, create a Statement instance in the following manner: stmt = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
The combination of a forward-only, read-only result set, with a fetch size of Integer.MIN_VALUE serves as a signal to the driver to stream result sets row-by-row. After this, any result sets created with the statement will be retrieved row-by-row. There are some caveats with this approach. You must read all of the rows in the result set (or close it) before you can issue any other queries on the connection, or an exception will be thrown. The earliest the locks these statements hold can be released (whether they be MyISAM table-level locks or row-level locks in some other storage engine such as InnoDB) is when the statement completes. If the statement is within scope of a transaction, then locks are released when the transaction completes (which implies that the statement needs to complete first). As with most other databases, statements are not complete until all the results pending on the statement are read or the active result set for the statement is closed. Therefore, if using streaming results, process them as quickly as possible if you want to maintain concurrent access to the tables referenced by the statement producing the result set.
通过联合使用forward-only,read-only resultSet和fetchsize值为Integer.MIN_VALUE作为驱动一行行获取结果流的信号。设置完以后,所有statement创建的resultSet将会一行行的获取结果集。
参考文献:
【1】 https://wiki.apache.org/solr/DataImportHandlerFaq
【2】http://dev.mysql.com/doc/connector-j/en/connector-j-reference-implementation-notes.html
使用solr的DIHandler 构建mysql大表全量索引,内存溢出问题的解决方法的更多相关文章
- Mysql备份系列(3)--innobackupex备份mysql大数据(全量+增量)操作记录
在日常的linux运维工作中,大数据量备份与还原,始终是个难点.关于mysql的备份和恢复,比较传统的是用mysqldump工具,今天这里推荐另一个备份工具innobackupex.innobacku ...
- [记录]一则清理MySQL大表以释放磁盘空间的案例
一则清理MySQL大表以释放磁盘空间的案例 一.基本情况: 1.dbtest库554G,先清理st_online_time_away_ds(37G)表的数据,保留半年的数据: 1)删除的数据:sele ...
- 优秀后端架构师必会知识:史上最全MySQL大表优化方案总结
本文原作者“ manong”,原创发表于segmentfault,原文链接:segmentfault.com/a/1190000006158186 1.引言 MySQL作为开源技术的代表作之一,是 ...
- MySQL 大表优化方案(长文)
当MySQL单表记录数过大时,增删改查性能都会急剧下降,可以参考以下步骤来优化: 单表优化 除非单表数据未来会一直不断上涨,否则不要一开始就考虑拆分,拆分会带来逻辑.部署.运维的各种复杂度,一般以整型 ...
- 从云数据迁移服务看MySQL大表抽取模式
摘要:MySQL JDBC抽取到底应该采用什么样的方式,且听小编给你娓娓道来. 小编最近在云上的一个迁移项目中被MySQL抽取模式折磨的很惨.一开始爆内存被客户怼,再后来迁移效率低下再被怼.MySQL ...
- mysql大表myisam的导入
在my.cnf中增大以下参数 myisam_sort_buffer_size = 1024Mtmp_table_size = 256M tmpdir = /home/tmpmyisam_max_sor ...
- MySql数据表设计,索引优化,SQL优化,其他数据库
MySql数据表设计,索引优化,SQL优化,其他数据库 1.数据表设计 1.1数据类型 1.2避免空值 1.3text类型优化 2.索引优化 2.1索引分类 2.2索引优化 3.SQL优化 3.1分批 ...
- 【small项目】MySQL第二天早上第一次连接超时报错,解决方法com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
MySQL第二天早上第一次连接超时报错,解决方法com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link ...
- golang github.com/go-sql-driver/mysql 遇到的数据库,设置库设计不合理的解决方法
golang github.com/go-sql-driver/mysql 遇到的数据库,设置库设计不合理的解决方法,查询中报了以下这个错 Scan error on column index 2: ...
随机推荐
- Pycharm 的安装
一. Windows 安装 汉化 破解补丁激活 下载 `https://pan.baidu.com/s/1qjI9uHaw0x374rwu6H8djA` 并将 JetbrainsCrack-2.8-r ...
- C# 利用反射和特性 来做一些事情
特性代码: [AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = false)] public clas ...
- 8种提升 ASP.NET Web API 性能的方法
- Sandbox
Sandbox Contents Overview Design principles Sandbox windows architecture The broker process The targ ...
- node.js状态码
100——客户必须继续发出请求101——客户要求服务器根据请求转换HTTP协议版本200——交易成功201——提示知道新文件的URL202——接受和处理.但处理未完成203——返回信息不确定或不完整2 ...
- mescroll报错
1.Cannot read property 'insertBefore' of null:说明你的容器id未找到,应确认你的容器id名与你NEW的容器名一致:
- Safe and efficient allocation of memory
Aspects of the present invention are directed at centrally managing the allocation of memory to exec ...
- 使用DbUtils实现增删改查——ResultSetHandler 接口的实现类
在上一篇文章中<使用DbUtils实现增删改查>,发现运行runner.query()这行代码时.须要自己去处理查询到的结果集,比較麻烦.这行代码的原型是: public Object q ...
- PHP开发常规安全问题总结
文章来源:PHP开发学习门户 地址:http://www.phpthinking.com/archives/575 php给了开发人员极大的灵活性,可是这也为安全问题带来了潜在的隐患,最近须要总结一下 ...
- 运行maven项目出现的报错
java问题:严重: Error configuring application listener of class org.springframework.web.context.Cont 解决方案 ...