使用Ruby处理大型CSV文件
处理大型文件是一种内存密集型操作,可能导致服务器耗尽RAM内存并交换到磁盘。让我们看一下使用Ruby处理CSV文件的几种方法,并测量内存消耗和速度性能。
Prepare CSV data sample
Before we start, let's prepare a CSV file data.csv
with 1 million rows (~ 75 MB) to use in tests.
require 'csv'
require_relative './helpers' headers = ['id', 'name', 'email', 'city', 'street', 'country'] name = "Pink Panther"
email = "pink.panther@example.com"
city = "Pink City"
street = "Pink Road"
country = "Pink Country" print_memory_usage do
print_time_spent do
CSV.open('data.csv', 'w', write_headers: true, headers: headers) do |csv|
1_000_000.times do |i|
csv << [i, name, email, city, street, country]
end
end
end
end
Memory used and time spent
This script above requires the helpers.rb
script which defines two helper methods for measuring and printing out the memory used and time spent.
require 'benchmark' def print_memory_usage
memory_before = `ps -o rss= -p #{Process.pid}`.to_i
yield
memory_after = `ps -o rss= -p #{Process.pid}`.to_i puts "Memory: #{((memory_after - memory_before) / 1024.0).round(2)} MB"
end def print_time_spent
time = Benchmark.realtime do
yield
end puts "Time: #{time.round(2)}"
end
The results to generate the CSV file are:
$ ruby generate_csv.rb
Time: 5.17
Memory: 1.08 MB
Output can vary between machines, but the point is that when building the CSV file, the Ruby process did not spike in memory usage because the garbage collector (GC) was reclaiming the used memory. The memory increase of the process is about 1MB, and it created a CSV file with size of 75 MB.
$ ls -lah data.csv
-rw-rw-r-- 1 dalibor dalibor 75M Mar 29 00:34 data.csv
Reading CSV from a file at once (CSV.read)
Let's build a CSV object from a file (data.csv
) and iterate with the following script:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
csv = CSV.read('data.csv', headers: true)
sum = 0 csv.each do |row|
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
The results are:
$ ruby parse1.rb
Sum: 499999500000
Time: 19.84
Memory: 920.14 MB
Important to note here is the big memory spike to 920 MB. That is because we build the whole CSV object in memory. That causes lots of String objects to be created by the CSV library and the used memory is much more higher than the actual size of the CSV file.
Parsing CSV from in memory String (CSV.parse)
Let's build a CSV object from a content in memory and iterate with the following script:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
content = File.read('data.csv')
csv = CSV.parse(content, headers: true)
sum = 0 csv.each do |row|
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
The results are:
$ ruby parse2.rb
Sum: 499999500000
Time: 21.71
Memory: 1003.69 MB
As we can see from the results, the memory increase is about the memory increase from the previous example plus the memory size of the file content that we read in memory (75MB).
Parsing CSV line by line from String in memory (CSV.new)
Let's now see what happens if we load the file content in a String and parse it line by line:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
content = File.read('data.csv')
csv = CSV.new(content, headers: true)
sum = 0 while row = csv.shift
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
The results are:
$ ruby parse3.rb
Sum: 499999500000
Time: 9.73
Memory: 74.64 MB
From the results we can see that the memory used is about the file size (75 MB) because the file content is loaded in memory and the processing time is about twice faster. This approach is useful when we have the content that we don't need to read it from a file and we just want to iterate over it line by line.
Parsing CSV file line by line from IO object
Can we do any better than the previous script? Yes, if we have the CSV content in a file. Let's use an IO file object directly:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
File.open('data.csv', 'r') do |file|
csv = CSV.new(file, headers: true)
sum = 0 while row = csv.shift
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
end
The results are:
$ ruby parse4.rb
Sum: 499999500000
Time: 9.88
Memory: 0.58 MB
In the last script we see less than 1 MB of memory increase. Time seems to be a very little slower compared to previous script because there is more IO involved. The CSV library has a built in mechanism for this, CSV.foreach
:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
sum = 0 CSV.foreach('data.csv', headers: true) do |row|
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
结果类似:
$ ruby parse5.rb
Sum: 499999500000
Time: 9.84
Memory: 0.53 MB
想象一下,您需要处理10GB或更大的大型CSV文件。决定使用最后一个策略似乎是显而易见的。
使用Ruby处理大型CSV文件的更多相关文章
- 建议42:使用pandas处理大型CSV文件
# -*- coding:utf-8 -*- ''' CSV 常用API 1)reader(csvfile[, dialect='excel'][, fmtparam]),主要用于CSV 文件的读取, ...
- Python 从大型csv文件中提取感兴趣的行
帮妹子处理一个2.xG 大小的 csv文件,文件太大,不宜一次性读入内存,可以使用open迭代器. with open(filename,'r') as file # 按行读取 for line in ...
- 109.大型的csv文件的处理方式
HttpResponse对象将会将响应的数据作为一个整体返回,此时如果数据量非常大的话,长时间浏览器没有得到服务器的响应,就会超过默认的超时时间,返回超时.而StreamingHttpResponse ...
- Django学习笔记之视图高级-CSV文件生成
生成CSV文件 有时候我们做的网站,需要将一些数据,生成有一个CSV文件给浏览器,并且是作为附件的形式下载下来.以下将讲解如何生成CSV文件. 生成小的CSV文件 这里将用一个生成小的CSV文件为例. ...
- Django生成CSV文件
1.生成CSV文件 有时候我们做的网站,需要将一些数据,生成有一个CSV文件给浏览器,并且是作为附件的形式下载下来.以下将讲解如何生成CSV文件. 2.生成小的CSV文件 这里将用一个生成小的CSV文 ...
- POI以SAX方式解析Excel2007大文件(包含空单元格的处理) Java生成CSV文件实例详解
http://blog.csdn.net/l081307114/article/details/46009015 http://www.cnblogs.com/dreammyle/p/5458280. ...
- [Python]-pandas模块-CSV文件读写
Pandas 即Python Data Analysis Library,是为了解决数据分析而创建的第三方工具,它不仅提供了丰富的数据模型,而且支持多种文件格式处理,包括CSV.HDF5.HTML 等 ...
- CSV文件分割与列异常处理的python脚本
csv文件通常存在如下问题: 1. 文件过大(需要进行文件分割)2. 列异常(列不一致,如元数据列为10列,但csv文件有些行是11列,或者4列)本脚本用于解决此问题. #coding=utf-8 ' ...
- 用opencsv文件读写CSV文件
首先明白csv文件长啥样儿: 用excel打开就变成表格了,看不到细节 推荐用其它简单粗暴一点儿的编辑器,比如Notepad++, csv文件内容如下: csv文件默认用逗号分隔各列. 有了基础的了解 ...
随机推荐
- centos 7创建ss服务(方式二)
一:安装pip yum install python-pip 如果没有python包则执行命令:yum -y install epel-release: 二:安装SS pip install shad ...
- 【XSY2707】snow 线段树 并查集
题目描述 有\(n\)个人和一条长度为\(t\)的线段,每个人还有一个工作范围(是一个区间).最开始整条线段都是白的.定义每个人的工作长度是这个人的工作范围中白色部分的长度(会随着线段改变而改变).每 ...
- python_sort(key=) 的使用
""" sort() 函数内含有 key 可以指定相关的函数来作为排序依据 比如这里指定每一项的索引为1 的元素作为排序依据 默认是以第一个作为排序依据 "&q ...
- 睡眠麻痹 CSP HSP
睡眠麻痹 CSP HSP 来源 https://www.zhihu.com/question/29666875/answer/65480583 俗名“鬼压床”.“鬼压身”或者“梦魇”的,学名叫睡眠麻痹 ...
- Java 枚举(enum) 详解7种常见的用法
Java 枚举(enum) 详解7种常见的用法 来源 https://blog.csdn.net/qq_27093465/article/details/52180865 JDK1.5引入了新的类型— ...
- MT【285】含参数函数绝对值的最大值
(浙江2013高考压轴题)已知$a\in R$,函数$f(x)=x^3-3x^2+3ax-3a+3$(2)当$x\in[0,2]$时,求$|f(x)|$的最大值. 分析:由题意$f^{'}(x)=3x ...
- 【hdu 4658】Integer Partition (无序分拆数、五边形数定理)
hdu 4658 Integer Partition 题意 n分拆成若干个正整数的和,每个正整数出现小于k次,分拆方案有多少.(t<=100,n<=1e5) 题解 之前写过一篇Partit ...
- Nginx+Tomcat-cluster构建
-----------ReProxy-------------------------Client-----------192.168.56.202 nginx 192.168.56.200 Tomc ...
- day15-集合
快捷键: 先定义name,age再利用快捷键生成,Alt+shift+s +c:空参 Alt+shift+s +o:有参 Alt+shift+s +r:set&get方法Ctrl+shift+ ...
- [SCOI2016]萌萌哒(倍增+并查集)
一个长度为n的大数,用S1S2S3...Sn表示,其中Si表示数的第i位,S1是数的最高位,告诉你一些限制条件,每个条件表示为四个数,l1,r1,l2,r2,即两个长度相同的区间,表示子串Sl1Sl1 ...