Scalding初探之一：基于Scala的Hadoop利器

把你从写繁琐的Map-reduce Job中解放出来，写分布式跟写本地程序没两样，Scala真真代表着先进生产力的方向啊。

好的，今天开始直播基于Scala的Scalding啦，循序渐进地看以下页面：

https://github.com/twitter/scalding#scalding

https://github.com/twitter/scalding/wiki/Getting-Started

https://github.com/willf/scalding_cookbook

看到scalding-cookbook的时候，可以开始尝试写比Word Count更酷的Scalding程序了

 import com.twitter.scalding._
 // input (tsv)  // 0 1 2 3 4 5 6  // 22 kinds of love nn2 io nn1  // 12 large green eyes jj jj nn2  //
 // output (tsv)  // 22 of kinds/nn2_love/nn1  // 12 green large/jj_eyes/nn2

 class contextCountJob(args : Args) extends Job(args) {    val inSchema = ('count, 'w1 ,'w2, 'w3, 'pos1, 'pos2, 'pos3)
   val outSchema = ('count, 'word, 'context)
   Tsv(args("input"),inSchema)      .mapTo(inSchema -> outSchema) {        parts : (String, String, String, String, String, String, String) => {          val (count, w1, w2, w3, pos1, pos2, pos3) = parts          val context = "%s/%s_%s/%s".format(w1,pos1,w3,pos3)   (count, w2, context)   }   }    .write(Tsv(args("output")))  }

比较糟糕的是Scala语言新潮到博客园插件都不支持。。。

http://docs.kiji.org/userguides/express/1.0.1/basic-scala-scalding/

http://sujitpal.blogspot.com/2012/08/scalding-for-impatient.html

https://github.com/sujitpal/hia-examples/tree/master/scala/scalding-impatient/src/main/scala/com/mycompany/impatient

https://github.com/twitter/scalding/wiki/Fields-based-API-Reference

https://github.com/twitter/scalding/wiki/Scalding-Sources

https://github.com/twitter/scalding/wiki/Field-rules

https://github.com/twitter/scalding/wiki/API-Reference

https://github.com/twitter/scalding/wiki

http://twitter.github.io/scalding/com/twitter/scalding/package.html

https://github.com/deanwampler/scalding-workshop

推荐Twitter公开课的PPT（此处应有翻墙）

Scalding: Twitter's Scala DSL for Hadoop/Cascading from johnynek

可以对比下其他Hadoop框架

Should I Use Scalding or Scoobi or Scrunch? from Hadoop_Summit

 

Introduction to Scalding and Monoids from Hugo Gävert

 

How LinkedIn Uses Scalding for Data Driven Product Development from Sasha Ovsankin

 

https://github.com/cloudera/crunch/tree/master/scrunch

http://www.slideshare.net/Hadoop_Summit/severs-june26-255pmroom210av2

http://blog.samibadawi.com/2012/03/hive-pig-scalding-scoobi-scrunch-and.html

http://www.quora.com/Apache-Hadoop/What-are-the-differences-between-Crunch-and-Cascading

http://stackoverflow.com/questions/9427974/what-is-the-most-mature-library-for-pipeline-building-in-java-scala-for-hadoop

https://github.com/echen/rosetta-scone

http://mcfunley.com/scalding-at-etsy (Slide 52)