We've seen the internals of MapReduce in the last post. Now we can make a little change to the WordCount and create a JAR for being executed by Hadoop. 
If we look at the result of the WordCount we ran before, the lines of the file are only split by space, and thus all other punctuation characters and symbols remain attached to the words and in some way "invalidate" the count. For example, we can see some of these wrong values:

KING 1
KING'S 1
KING. 5
King 6
King, 4
King. 2
King; 1

The word is always king but sometimes it appears in upper case, sometimes in lower case, or with a punctuation character after it. So we'd like to update the behaviour of the WordCount program to count all the occurrences of any word, aside from the punctuation and other symbols. In order to do so, we have to modify the code of the mapper, since it's here that we get the data from the file and split it. 
If we look at the code of mapper:

public static class TokenizerMapper extends Mapper<object, text,="" intwritable=""> {

      private final static IntWritable one = new IntWritable(1);
private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}

we see that the StringTokenizer takes the line as the parameter; before doing that, we remove all the symbols from the line using a RegExp that maps each of these symbols into a space:

String cleanLine = value.toString().toLowerCase().replaceAll("[_|$#<>\\[\\]\\*/\\\\,;,.\\-:()?!\"']", " ");

that simply says "if you see any of these character _, |, $, #, <, >, [, ], *, /, \, ,, ;, ., -, :, ,(, ), ?, !, ", ' transform it into a space". All the backslashes are needed for correctly escaping the characters. Then we trim the token to avoid empty tokens:

itr.nextToken().trim()

So now the code looks like this (the updates to the original file are printed in bold):

public static class TokenizerMapper extends Mapper<object, text,="" intwritable=""> {

 private final static IntWritable one = new IntWritable(1);
private Text word = new Text(); @Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
String cleanLine = value.toString().toLowerCase().replaceAll("[_|$#<>\\^=\\[\\]\\*/\\\\,;,.\\-:()?!\"']", " ");
StringTokenizer itr = new StringTokenizer(cleanLine);
while (itr.hasMoreTokens()) {
word.set(itr.nextToken().trim());
context.write(word, one);
}
}
}

The complete code of the class is available on my github repository.

We now want to create the JAR to be executed; to do it, we have to go to the output directory of our wordcount file (the directory that contains the .class files) and create a manifest filenamed manifest_file that contains something like this:

Manifest-Version: 1.0
Main-Class: samples.wordcount.WordCount

in which we tell the JAR which class to execute at startup. More details in Java official documentation. Note that there's no need to add classpath info, because the JAR will be run by Hadoop that already has a classpath that contains all needed libraries. 
Now we can launch the command:

$ jar cmf manifest_file wordcount.jar .

that creates a JAR named wordcount.jar that contains all classes starting from this directory (the . parameter) e using the content of manifest_file to create a Manifest. 
We can run the batch as we saw before. Looking at the result file we can check that there are no more symbols and punctuation characters; the occurrences of the word king is now the sum of all occurrences we found before:

king 20

which is what we were looking for.

from: http://andreaiacono.blogspot.com/2014/02/creating-jar-for-hadoop.html

为Hadoop创建JAR包文件Creating a JAR for Hadoop的更多相关文章

  1. 有引用外部jar包时(J2SE)生成jar文件

    一.工程没有引用外部jar包时(J2SE) 选中工程---->右键,Export...--->Java--->选择JAR file--->next-->选择jar fil ...

  2. jmeter,badboy,jar包文件 常数吞吐量计时器?

    badboy录制脚本 1.按f2 红色开始录制 URL输入:https://www.so.com/ 2.搜索框输入zxw 回车键搜索 3.选中关键字(刮例如zxw软件——>tools——> ...

  3. spring原理案例-基本项目搭建 02 spring jar包详解 spring jar包的用途

    Spring4 Jar包详解 SpringJava Spring AOP: Spring的面向切面编程,提供AOP(面向切面编程)的实现 Spring Aspects: Spring提供的对Aspec ...

  4. Eclipse用Runnable JAR file方式打jar包,并用该jar包进行二次开发

    目录: 1.eclipse创建Java项目(带jar包的) 2. eclipse用Export的Runnable JAR file方式打jar包(带jar包的) 打jar包 1)class2json1 ...

  5. 多个jar包合并成一个jar包(ant)

    https://blog.csdn.net/gzl003csdn/article/details/53539133 多个jar包合并成一个jar 使用Apache的Ant是一个基于Java的生成工具. ...

  6. spring boot:在项目中引入第三方外部jar包集成为本地jar包(spring boot 2.3.2)

    一,为什么要集成外部jar包? 不是所有的第三方库都会上传到mvnrepository, 这时我们如果想集成它的第三方库,则需要直接在项目中集成它们的jar包, 在操作上还是很简单的, 这里用luos ...

  7. 多个jar包合并成一个jar包的办法

    步骤: 1.将多个JAR包使用压缩软件打开,并将全包名的类拷贝到一个临时目录地下. 2.cmd命令到该临时目录下,此时会有很多.class文件,其中需要带完整包路径 3.执行 jar -cvfM te ...

  8. Ant打包可运行的Jar包(加入第三方jar包)

    本章介绍使用ant打包可运行的Jar包. 打包jar包最大的问题在于如何加入第三方jar包使得jar文件可以直接运行.以下用一个实例程序进行说明. 程序结构: 关键代码: package com.al ...

  9. 【Maven学习】Maven打包生成普通jar包、可运行jar包、包含所有依赖的jar包

    http://blog.csdn.net/u013177446/article/details/54134394 ******************************************* ...

随机推荐

  1. Dell服务器iDrac口默认账号密码和IP

    https://blog.csdn.net/artdao1987/article/details/79875528

  2. Web前端开发最佳实践(4):在页面中添加必要的meta信息

    meta标签放置在HTML页面的head中,主要用于标识网站.其中基本上包含了网站的一些描述信息,例如,简介.作者等.这些信息有助于搜索引擎更准确地识别网页的内容,也有助于第三方工具抓取网站基本信息. ...

  3. Java 中静态代码块初始化问题测试

    Java 中静态代码块初始化问题测试 原创 情况一:变量是 static final 修饰的"编译期常量",如 public static final String a = &qu ...

  4. Nginx 详细安装部署教程

    一.Nginx简介 Nginx是一个web服务器也可以用来做负载均衡及反向代理使用,目前使用最多的就是负载均衡,具体简介我就不介绍了百度一下有很多,下面直接进入安装步骤 二.Nginx安装 1.下载N ...

  5. sublimetext3-实用快捷键整理

    实用快捷键 Ctrl+Shift+P:打开命令面板Ctrl+P:搜索项目中的文件Ctrl+G:跳转到第几行Ctrl+W:关闭当前打开文件Ctrl+Shift+W:关闭所有打开文件Ctrl+Shift+ ...

  6. java浅拷贝和深拷贝(基础也是很重要的)

    对象的copy你兴许只是懵懂,或者是并没在意,来了解下吧. 对于的github基础代码https://github.com/chywx/JavaSE 最近学习c++,跟java很是相像,在慕课网学习c ...

  7. mybatis递归,一对多代码示例

    今天需要做一个功能,根据专业,有不同的章节,章节下面有对应的习题, 由于只有这么两级,可以不用使用递归,直接查询父集,之后foreach查询子集放入对应的list集合. 虽然实现了,感觉毕竟,太low ...

  8. 验证Xcode真伪的方法,来自苹果官网

    验证Xcode真伪的方法,来自苹果官网   Xcode的验证你的版本 2015年9月22日    注意:中文为有道翻译,看下验证方法即可.   我们最近将应用程序从应用程序商店,还建有Xcode的假冒 ...

  9. 【BZOJ 2946】 2946: [Poi2000]公共串 (SAM)

    2946: [Poi2000]公共串 Time Limit: 3 Sec  Memory Limit: 128 MBSubmit: 1063  Solved: 469 Description      ...

  10. Codeforces 992 E. Nastya and King-Shamans

    \(>Codeforces\space992 E. Nastya and King-Shamans<\) 题目大意 : 给你一个长度为 \(n\) 的序列,有 \(q\) 次操作,每一次操 ...