使用Ruby处理大型CSV文件
处理大型文件是一种内存密集型操作,可能导致服务器耗尽RAM内存并交换到磁盘。让我们看一下使用Ruby处理CSV文件的几种方法,并测量内存消耗和速度性能。
Prepare CSV data sample
Before we start, let's prepare a CSV file data.csv
with 1 million rows (~ 75 MB) to use in tests.
require 'csv'
require_relative './helpers' headers = ['id', 'name', 'email', 'city', 'street', 'country'] name = "Pink Panther"
email = "pink.panther@example.com"
city = "Pink City"
street = "Pink Road"
country = "Pink Country" print_memory_usage do
print_time_spent do
CSV.open('data.csv', 'w', write_headers: true, headers: headers) do |csv|
1_000_000.times do |i|
csv << [i, name, email, city, street, country]
end
end
end
end
Memory used and time spent
This script above requires the helpers.rb
script which defines two helper methods for measuring and printing out the memory used and time spent.
require 'benchmark' def print_memory_usage
memory_before = `ps -o rss= -p #{Process.pid}`.to_i
yield
memory_after = `ps -o rss= -p #{Process.pid}`.to_i puts "Memory: #{((memory_after - memory_before) / 1024.0).round(2)} MB"
end def print_time_spent
time = Benchmark.realtime do
yield
end puts "Time: #{time.round(2)}"
end
The results to generate the CSV file are:
$ ruby generate_csv.rb
Time: 5.17
Memory: 1.08 MB
Output can vary between machines, but the point is that when building the CSV file, the Ruby process did not spike in memory usage because the garbage collector (GC) was reclaiming the used memory. The memory increase of the process is about 1MB, and it created a CSV file with size of 75 MB.
$ ls -lah data.csv
-rw-rw-r-- 1 dalibor dalibor 75M Mar 29 00:34 data.csv
Reading CSV from a file at once (CSV.read)
Let's build a CSV object from a file (data.csv
) and iterate with the following script:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
csv = CSV.read('data.csv', headers: true)
sum = 0 csv.each do |row|
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
The results are:
$ ruby parse1.rb
Sum: 499999500000
Time: 19.84
Memory: 920.14 MB
Important to note here is the big memory spike to 920 MB. That is because we build the whole CSV object in memory. That causes lots of String objects to be created by the CSV library and the used memory is much more higher than the actual size of the CSV file.
Parsing CSV from in memory String (CSV.parse)
Let's build a CSV object from a content in memory and iterate with the following script:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
content = File.read('data.csv')
csv = CSV.parse(content, headers: true)
sum = 0 csv.each do |row|
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
The results are:
$ ruby parse2.rb
Sum: 499999500000
Time: 21.71
Memory: 1003.69 MB
As we can see from the results, the memory increase is about the memory increase from the previous example plus the memory size of the file content that we read in memory (75MB).
Parsing CSV line by line from String in memory (CSV.new)
Let's now see what happens if we load the file content in a String and parse it line by line:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
content = File.read('data.csv')
csv = CSV.new(content, headers: true)
sum = 0 while row = csv.shift
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
The results are:
$ ruby parse3.rb
Sum: 499999500000
Time: 9.73
Memory: 74.64 MB
From the results we can see that the memory used is about the file size (75 MB) because the file content is loaded in memory and the processing time is about twice faster. This approach is useful when we have the content that we don't need to read it from a file and we just want to iterate over it line by line.
Parsing CSV file line by line from IO object
Can we do any better than the previous script? Yes, if we have the CSV content in a file. Let's use an IO file object directly:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
File.open('data.csv', 'r') do |file|
csv = CSV.new(file, headers: true)
sum = 0 while row = csv.shift
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
end
The results are:
$ ruby parse4.rb
Sum: 499999500000
Time: 9.88
Memory: 0.58 MB
In the last script we see less than 1 MB of memory increase. Time seems to be a very little slower compared to previous script because there is more IO involved. The CSV library has a built in mechanism for this, CSV.foreach
:
require_relative './helpers'
require 'csv' print_memory_usage do
print_time_spent do
sum = 0 CSV.foreach('data.csv', headers: true) do |row|
sum += row['id'].to_i
end puts "Sum: #{sum}"
end
end
结果类似:
$ ruby parse5.rb
Sum: 499999500000
Time: 9.84
Memory: 0.53 MB
想象一下,您需要处理10GB或更大的大型CSV文件。决定使用最后一个策略似乎是显而易见的。
使用Ruby处理大型CSV文件的更多相关文章
- 建议42:使用pandas处理大型CSV文件
# -*- coding:utf-8 -*- ''' CSV 常用API 1)reader(csvfile[, dialect='excel'][, fmtparam]),主要用于CSV 文件的读取, ...
- Python 从大型csv文件中提取感兴趣的行
帮妹子处理一个2.xG 大小的 csv文件,文件太大,不宜一次性读入内存,可以使用open迭代器. with open(filename,'r') as file # 按行读取 for line in ...
- 109.大型的csv文件的处理方式
HttpResponse对象将会将响应的数据作为一个整体返回,此时如果数据量非常大的话,长时间浏览器没有得到服务器的响应,就会超过默认的超时时间,返回超时.而StreamingHttpResponse ...
- Django学习笔记之视图高级-CSV文件生成
生成CSV文件 有时候我们做的网站,需要将一些数据,生成有一个CSV文件给浏览器,并且是作为附件的形式下载下来.以下将讲解如何生成CSV文件. 生成小的CSV文件 这里将用一个生成小的CSV文件为例. ...
- Django生成CSV文件
1.生成CSV文件 有时候我们做的网站,需要将一些数据,生成有一个CSV文件给浏览器,并且是作为附件的形式下载下来.以下将讲解如何生成CSV文件. 2.生成小的CSV文件 这里将用一个生成小的CSV文 ...
- POI以SAX方式解析Excel2007大文件(包含空单元格的处理) Java生成CSV文件实例详解
http://blog.csdn.net/l081307114/article/details/46009015 http://www.cnblogs.com/dreammyle/p/5458280. ...
- [Python]-pandas模块-CSV文件读写
Pandas 即Python Data Analysis Library,是为了解决数据分析而创建的第三方工具,它不仅提供了丰富的数据模型,而且支持多种文件格式处理,包括CSV.HDF5.HTML 等 ...
- CSV文件分割与列异常处理的python脚本
csv文件通常存在如下问题: 1. 文件过大(需要进行文件分割)2. 列异常(列不一致,如元数据列为10列,但csv文件有些行是11列,或者4列)本脚本用于解决此问题. #coding=utf-8 ' ...
- 用opencsv文件读写CSV文件
首先明白csv文件长啥样儿: 用excel打开就变成表格了,看不到细节 推荐用其它简单粗暴一点儿的编辑器,比如Notepad++, csv文件内容如下: csv文件默认用逗号分隔各列. 有了基础的了解 ...
随机推荐
- codeforces 600E . Lomsat gelral (线段树合并)
You are given a rooted tree with root in vertex 1. Each vertex is coloured in some colour. Let's cal ...
- Codeforces 1045. A. Last chance(网络流 + 线段树优化建边)
题意 给你 \(n\) 个武器,\(m\) 个敌人,问你最多消灭多少个敌人,并输出方案. 总共有三种武器. SQL 火箭 - 能消灭给你集合中的一个敌人 \(\sum |S| \le 100000\) ...
- [CF977F]Consecutive Subsequence
题目描述 You are given an integer array of length n. You have to choose some subsequence of this array o ...
- 【nginx】nginx配置文件结构,内置变量及参数调优
Nginx的配置文件是一个纯文本文件,它一般位于Nginx安装目录的conf目录下,整个配置文件是以block的形式组织的.每个block一般以一个大括号“{”来表示.block 可以分为几个层次,整 ...
- 牛客寒假算法基础集训营3处女座和小姐姐(三) (数位dp)
链接:https://ac.nowcoder.com/acm/contest/329/G来源:牛客网 时间限制:C/C++ 1秒,其他语言2秒 空间限制:C/C++ 262144K,其他语言52428 ...
- 假如你不小心干掉了系统,你该怎么办?(一次手贱的记录 ~ Ubuntu and Python3.6)
前言 多年未犯低级错误,今天犯了个不大不小的错误,记录下生活点滴吧 今天早上脑海里想了下,如果电脑挂了我要备份哪些东西?然后中午休息的时候就列了一下,没想到晚上就悲剧了... 这个是中午写的: ## ...
- Anaconda For Linux (附C#交互式编程的引入)
汇总系列:https://www.cnblogs.com/dunitian/p/4822808.html#ai Jupyter美化: https://www.cnblogs.com/dotnetcra ...
- Spring 整合 Hibernate 时启用二级缓存实例详解
写在前面: 1. 本例使用 Hibernate3 + Spring3: 2. 本例的查询使用了 HibernateTemplate: 1. 导入 ehcache-x.x.x.jar 包: 2. 在 a ...
- Vim保存时权限不足
保存时权限不足,由于打开时忘记在命令前添加sudo.我们并不需要放弃修改,从新以root权限打开 解决方案 命令模式使用:w !sudo tee %提权,保存
- 洛谷P4248 差异
题意:求所有后缀两两之间的最长公共前缀的长度之和. 解:这道题让我发现了一个奇妙的性质:所有后缀两两最长公共前缀长度之和 和 所有前缀两两最长公共后缀之和的值是相等的,但是每一组公共前/后缀是不同的. ...