使用mongify将sqlserver数据导入到mongodb

最近需要将sqlserver数据导入到mongodb中，在github上搜了一圈，发现两个项目有点适合

1. mongify
2. sql2mongodb

先试了下sql2mongodb(有个好名字果然有好处啊), 可是，but，但是，这东西好像并不没有名字取得好，试了下感觉不靠谱，弃坑，逃~~~

说点题外话，sql2server以来nodejs， mongify依赖ruby， 这两个库都是我最讨厌的，暂时没有理由。

好了，现在咱专注mongify吧

先看官网介绍：

Mongify是一个将数据从sql数据库转换到mongoDB的ruby工具。支持MySQL，PostgreSQL，SQLite，Oracle，SQLServer和DB2（基本上ActiveRecord内置的内容），支持任何版本的MongoDB。

但是，缺点是作者声称只在MySql和SQLite测试过，对于我现在需要的sqlserver，心中一群在奔腾。。。

安装

使用的环境是ubuntu12.04，该版本软件仓库中的ruby是1.8版本的，需要ruby2.1版本以上才行

1. 省懒，直接加ppa吧, 这里可以随便安装到2.3版本
   
   From the travis-cli installation instructions for Ubuntu, the Brightbox Ruby NG(NextGeneration) ppa:

$ sudo apt-get install python-software-properties
$ sudo apt-add-repository ppa:brightbox/ruby-ng
$ sudo apt-get update
$ sudo apt-get install ruby2.1 ruby-switch
$ sudo ruby-switch --set ruby2.1

顺便注明下gem的使用吧

ubuntu@aws:~$ gem --help
RubyGems is a sophisticated package manager for Ruby.  This is a
basic help message containing pointers to more information.

  Usage:
    gem -h/--help
    gem -v/--version
    gem command [arguments...] [options...]

  Examples:
    gem install rake
    gem list --local
    gem build package.gemspec
    gem help install

  Further help:
    gem help commands            list all 'gem' commands
    gem help examples            show some examples of usage
    gem help gem_dependencies    gem dependencies file guide
    gem help platforms           gem platforms guide
    gem help <COMMAND>           show help on COMMAND
                                   (e.g. 'gem help install')
    gem server                   present a web page at
                                 http://localhost:8808/
                                 with info about installed gems
  Further information:
    http://guides.rubygems.org

好，那我们安装mongify吧gem install mongify

经过一轮踩坑，还是关注下mongify目录结构吧 其中有一个文件mongify.gemspec

$:.push File.expand_path("../lib", __FILE__)
require "mongify/version"

Gem::Specification.new do |s|
  s.name        = "mongify"
  s.version     = Mongify::VERSION
  s.platform    = Gem::Platform::RUBY
  s.authors     = ["Andrew Kalek"]
  s.email       = ["andrew.kalek@anlek.com"]
  s.homepage    = "http://mongify.com"
  s.summary     = %q{Translate your SQL data to MongoDB with ease}
  s.description = %q{Mongify allows you to map your sql data into a mongodb document database with a simple DSL.}
  s.required_ruby_version = ">= 1.8.7"

  s.add_runtime_dependency('activerecord', ">= 4.2", "< 5.0")
  s.add_runtime_dependency('activesupport', ">= 4.2", "< 5.0")
  s.add_runtime_dependency('mongo', "= 1.12.5")
  s.add_runtime_dependency('bson', "= 1.12.5")
  s.add_runtime_dependency('bson_ext', "= 1.12.5") unless RUBY_PLATFORM == 'java'
  s.add_runtime_dependency('highline', '= 1.7.8')

  s.add_development_dependency('rspec', '~> 2.0')
  s.add_development_dependency('rspec-collection_matchers', '~> 1.0')
  s.add_development_dependency('cucumber', '>= 0.10')
  s.add_development_dependency('mocha', '>= 0.9.8')
  s.add_development_dependency('yard', '>= 0.8')
  s.add_development_dependency('sqlite3', '>= 1.3')
  s.add_development_dependency('pg', '>= 0.17')
  s.add_development_dependency('mysql2', '>= 0.4')
  s.add_development_dependency('watchr', '>= 0.6')
  s.add_development_dependency('rake')
  s.add_development_dependency('jazz_fingers')

  s.files         = `git ls-files`.split("\n")
  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
  s.require_paths = ["lib"]

  s.extra_rdoc_files = [
    "CHANGELOG.md",
    "README.rdoc"
  ]
  s.rdoc_options = ["--title", "Mongify -- SQL db to Mongo db converter", "--main", "README", "--line-numbers", "--inline-source"]
end

可以看到这里指定了依赖，对版本也有要求。

如果已经安装过mongify1.3.1版本的话，导出sqlserver数据库时将会发现如果要支持sqlserver数据库还需要安装activerecord-sqlserver-adapter

而在activerecord-sqlserver-adapter官网有这样的描述

The SQL Server adapter for ActiveRecord v5.0 using SQL Server 2012 or higher.

Interested in older versions? We follow a rational versioning policy that tracks Rails. That means that our 5.0.x version of the adapter is only for the latest 5.0 version of Rails. If you need the adapter for SQL Server 2008 or 2005, you are still in the right spot. Just install the latest 3.2.x to 4.1.x version of the adapter that matches your Rails version. We also have stable branches for each major/minor release of ActiveRecord.

由于我的sqlserver是2008版本的，也就是说我最高可以使用4.1.x版本，并且最好有配套的ActiveRecord

所以我选择了1.2.4版本的Mongify，可以看下这个版本的依赖

  s.add_dependency('activerecord', "~> 3.2")
  s.add_dependency('activesupport', "~> 3.2")
  s.add_dependency('mongo', "~> 1.10.2")
  s.add_dependency('bson', "~> 1.10.2")
  s.add_dependency('bson_ext', "~> 1.10.2") unless RUBY_PLATFORM == 'java'
  s.add_dependency('highline', '>= 1.6.1')

对应的可以满足3.2.x-4.1.x这个要求，那么就是他了

重新安装Mongify

gem install mongify -v 1.2.4

根据上面的依赖，会安装对应版本软件再安装activerecord-sqlserver-adapter

gem install activerecord-sqlserver-adapter -v 3.2.17

创建配置文件

为了让Mongify正常工作，它需要知道数据库的位置及连接信息。

构建配置文件就像这样简单： (对于我们，adapter需要改成sqlserver)

sql_connection do
  adapter     "mysql"
  host        "localhost"
  username    "root"
  password    "passw0rd"
  database    "my_database"
  batch_size  10000           # This is defaulted to 10000 but in case you want to make that smaller (on lower RAM machines)
end

mongodb_connection do
  host        "localhost"
  database    "my_database"
end

检查配置文件

mongify check database.config

生成转换文件

如果您的数据库庞大而复杂，手工编写转换文件可能会有太多的工作。 Mongify的可以直接使用命令生成：

mongify translation database.config

可以使用重定向将配置输出到文件

mongify translation database.config > translation_file.rb

最后将sql数据导入到mongodb中

mongify process database.config translation_file.rb

导出时每次读取10000条数据，以进度条显示进度

Copying playsrc (261/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:03
Copying playsrc (262/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:03
Copying playsrc (263/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:02
Copying playsrc (264/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:02
Copying playsrc (265/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:03
Copying playsrc (266/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:03
Copying playsrc (267/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:02
Copying playsrc (268/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:02
Copying playsrc (269/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:03
Copying playsrc (270/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:03
Copying playsrc (271/607):            (10000/10000) 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| Time: 00:00:02

附录Mongify的命令，总共就四个主要命令，以上用过了三个

Usage: mongify command database_config [database_translation.rb]

Commands:
  "check" or "ck"           >> Checks connection for sql and no_sql databases [configuration_file]
  "process" or "pr"         >> Takes a translation and process it to mongodb [configuration_file, translation_file]
  "sync" or "sy"            >> Takes a translation and process it to mongodb, only syncs (insert/update) new or updated records based on the updated_at column [configuration_file, translation_file]
  "translation" or "tr"     >> Outputs a translation file from a sql connection [configuration_file]

Examples:

  mongify translation datbase.config
  mongify tr database.config
  mongify check database.config
  mongify process database.config database_translation.rb
  mongify sync database.config database_translation.rb

Common options:
    -h, --help                       Show this message
    -v, --version                    Show version