Linux巩固记录（5） hadoop 2.7.4下自己编译代码并运行MapReduce程序

程序代码为 ~\hadoop-2.7.4\share\hadoop\mapreduce\sources\hadoop-mapreduce-examples-2.7.4-sources\org\apache\hadoop\examples\WordCount.java

第一次删除了package

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

  public static class TokenizerMapper

       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);

    private Text word = new Text();

    public void map(Object key, Text value, Context context

                    ) throws IOException, InterruptedException {

      StringTokenizer itr = new StringTokenizer(value.toString());

      while (itr.hasMoreTokens()) {

        word.set(itr.nextToken());

        context.write(word, one);

      }

    }

  }

  public static class IntSumReducer

       extends Reducer<Text,IntWritable,Text,IntWritable> {

    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,

                       Context context

                       ) throws IOException, InterruptedException {

      int sum = 0;

      for (IntWritable val : values) {

        sum += val.get();

      }

      result.set(sum);

      context.write(key, result);

    }

  }

  public static void main(String[] args) throws Exception {

    Configuration conf = new Configuration();

    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

    if (otherArgs.length < 2) {

      System.err.println("Usage: wordcount <in> [<in>...] <out>");

      System.exit(2);

    }

    Job job = Job.getInstance(conf, "word count");

    job.setJarByClass(WordCount.class);

    job.setMapperClass(TokenizerMapper.class);

    job.setCombinerClass(IntSumReducer.class);

    job.setReducerClass(IntSumReducer.class);

    job.setOutputKeyClass(Text.class);

    job.setOutputValueClass(IntWritable.class);

    for (int i = 0; i < otherArgs.length - 1; ++i) {

      FileInputFormat.addInputPath(job, new Path(otherArgs[i]));

    }

    FileOutputFormat.setOutputPath(job,

      new Path(otherArgs[otherArgs.length - 1]));

    System.exit(job.waitForCompletion(true) ? 0 : 1);

  }

}

此程序需要下面三个jar包才能编译通过

[root@master classes]# tree /home/jars/

/home/jars/

├── commons-cli-1.4.jar

├── hadoop-common-2.7.4.jar

└── hadoop-mapreduce-client-core-2.7.4.jar

执行过程及结果如下

[root@master classes]#

[root@master classes]# pwd

/home/classes

[root@master classes]# tree

.

0 directories, 0 files

[root@master classes]# tree /home/javaFile/

/home/javaFile/

└── WordCount.java

0 directories, 1 file

[root@master classes]# tree /home/jars/

/home/jars/

├── commons-cli-1.4.jar

├── hadoop-common-2.7.4.jar

└── hadoop-mapreduce-client-core-2.7.4.jar

0 directories, 3 files

[root@master classes]# javac -classpath .:/home/jars/* -d /home/classes/ /home/javaFile/WordCount.java

[root@master classes]# tree

.

├── WordCount.class

├── WordCount$IntSumReducer.class

└── WordCount$TokenizerMapper.class

0 directories, 3 files

[root@master classes]# jar -cvf wordc.jar ./*.class

added manifest

adding: WordCount.class(in = 1907) (out= 1040)(deflated 45%)

adding: WordCount$IntSumReducer.class(in = 1739) (out= 742)(deflated 57%)

adding: WordCount$TokenizerMapper.class(in = 1736) (out= 753)(deflated 56%)

[root@master classes]# tree

.

├── wordc.jar

├── WordCount.class

├── WordCount$IntSumReducer.class

└── WordCount$TokenizerMapper.class

0 directories, 4 files

[root@master classes]# /home/hadoop-2.7.4/bin/hadoop jar /home/classes/wordc.jar WordCount /hdfs-input.txt /result-self-compile

17/09/02 02:11:45 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.80:8032

17/09/02 02:11:47 INFO input.FileInputFormat: Total input paths to process : 1

17/09/02 02:11:47 INFO mapreduce.JobSubmitter: number of splits:1

17/09/02 02:11:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504320356950_0010

17/09/02 02:11:47 INFO impl.YarnClientImpl: Submitted application application_1504320356950_0010

17/09/02 02:11:47 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504320356950_0010/

17/09/02 02:11:47 INFO mapreduce.Job: Running job: job_1504320356950_0010

17/09/02 02:11:56 INFO mapreduce.Job: Job job_1504320356950_0010 running in uber mode : false

17/09/02 02:11:56 INFO mapreduce.Job:  map 0% reduce 0%

17/09/02 02:12:02 INFO mapreduce.Job:  map 100% reduce 0%

17/09/02 02:12:09 INFO mapreduce.Job:  map 100% reduce 100%

17/09/02 02:12:09 INFO mapreduce.Job: Job job_1504320356950_0010 completed successfully

17/09/02 02:12:10 INFO mapreduce.Job: Counters: 49

    File System Counters

        FILE: Number of bytes read=118

        FILE: Number of bytes written=241697

        FILE: Number of read operations=0

        FILE: Number of large read operations=0

        FILE: Number of write operations=0

        HDFS: Number of bytes read=174

        HDFS: Number of bytes written=76

        HDFS: Number of read operations=6

        HDFS: Number of large read operations=0

        HDFS: Number of write operations=2

    Job Counters

        Launched map tasks=1

        Launched reduce tasks=1

        Data-local map tasks=1

        Total time spent by all maps in occupied slots (ms)=3745

        Total time spent by all reduces in occupied slots (ms)=4081

        Total time spent by all map tasks (ms)=3745

        Total time spent by all reduce tasks (ms)=4081

        Total vcore-milliseconds taken by all map tasks=3745

        Total vcore-milliseconds taken by all reduce tasks=4081

        Total megabyte-milliseconds taken by all map tasks=3834880

        Total megabyte-milliseconds taken by all reduce tasks=4178944

    Map-Reduce Framework

        Map input records=6

        Map output records=12

        Map output bytes=118

        Map output materialized bytes=118

        Input split bytes=98

        Combine input records=12

        Combine output records=9

        Reduce input groups=9

        Reduce shuffle bytes=118

        Reduce input records=9

        Reduce output records=9

        Spilled Records=18

        Shuffled Maps =1

        Failed Shuffles=0

        Merged Map outputs=1

        GC time elapsed (ms)=155

        CPU time spent (ms)=1430

        Physical memory (bytes) snapshot=299466752

        Virtual memory (bytes) snapshot=4159479808

        Total committed heap usage (bytes)=141385728

    Shuffle Errors

        BAD_ID=0

        CONNECTION=0

        IO_ERROR=0

        WRONG_LENGTH=0

        WRONG_MAP=0

        WRONG_REDUCE=0

    File Input Format Counters

        Bytes Read=76

    File Output Format Counters

        Bytes Written=76

[root@master classes]# /home/hadoop-2.7.4/bin/hadoop fs -ls /

Found 3 items

-rw-r--r--   2 root supergroup         76 2017-09-02 00:57 /hdfs-input.txt

drwxr-xr-x   - root supergroup          0 2017-09-02 02:12 /result-self-compile

drwx------   - root supergroup          0 2017-09-02 02:11 /tmp

[root@master classes]#

[root@master classes]#

第二次没有删除package

package org.apache.hadoop.examples;

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {

  public static class TokenizerMapper

       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);

    private Text word = new Text();

    public void map(Object key, Text value, Context context

                    ) throws IOException, InterruptedException {

      StringTokenizer itr = new StringTokenizer(value.toString());

      while (itr.hasMoreTokens()) {

        word.set(itr.nextToken());

        context.write(word, one);

      }

    }

  }

  public static class IntSumReducer

       extends Reducer<Text,IntWritable,Text,IntWritable> {

    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,

                       Context context

                       ) throws IOException, InterruptedException {

      int sum = 0;

      for (IntWritable val : values) {

        sum += val.get();

      }

      result.set(sum);

      context.write(key, result);

    }

  }

  public static void main(String[] args) throws Exception {

    Configuration conf = new Configuration();

    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

    if (otherArgs.length < 2) {

      System.err.println("Usage: wordcount <in> [<in>...] <out>");

      System.exit(2);

    }

    Job job = Job.getInstance(conf, "word count");

    job.setJarByClass(WordCount.class);

    job.setMapperClass(TokenizerMapper.class);

    job.setCombinerClass(IntSumReducer.class);

    job.setReducerClass(IntSumReducer.class);

    job.setOutputKeyClass(Text.class);

    job.setOutputValueClass(IntWritable.class);

    for (int i = 0; i < otherArgs.length - 1; ++i) {

      FileInputFormat.addInputPath(job, new Path(otherArgs[i]));

    }

    FileOutputFormat.setOutputPath(job,

      new Path(otherArgs[otherArgs.length - 1]));

    System.exit(job.waitForCompletion(true) ? 0 : 1);

  }

}

[root@master classes]#

[root@master classes]# tree

.

0 directories, 0 files

[root@master classes]# javac -classpath .:/home/jars/* -d /home/classes/ /home/javaFile/WordCount.java

[root@master classes]# tree

.

└── org

    └── apache

        └── hadoop

            └── examples

                ├── WordCount.class

                ├── WordCount$IntSumReducer.class

                └── WordCount$TokenizerMapper.class

4 directories, 3 files

[root@master classes]# jar -cvf wordcount.jar ./*

added manifest

adding: org/(in = 0) (out= 0)(stored 0%)

adding: org/apache/(in = 0) (out= 0)(stored 0%)

adding: org/apache/hadoop/(in = 0) (out= 0)(stored 0%)

adding: org/apache/hadoop/examples/(in = 0) (out= 0)(stored 0%)

adding: org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out= 764)(deflated 57%)

adding: org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1793) (out= 749)(deflated 58%)

adding: org/apache/hadoop/examples/WordCount.class(in = 1988) (out= 1050)(deflated 47%)

[root@master classes]# /home/hadoop-2.7.4/bin/hadoop jar /home/classes/wordcount.jar org.apache.hadoop.examples.WordCount /hdfs-input.txt /result-package

17/09/02 02:20:41 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.80:8032

17/09/02 02:20:43 INFO input.FileInputFormat: Total input paths to process : 1

17/09/02 02:20:43 INFO mapreduce.JobSubmitter: number of splits:1

17/09/02 02:20:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504320356950_0011

17/09/02 02:20:43 INFO impl.YarnClientImpl: Submitted application application_1504320356950_0011

17/09/02 02:20:43 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504320356950_0011/

17/09/02 02:20:43 INFO mapreduce.Job: Running job: job_1504320356950_0011

17/09/02 02:20:51 INFO mapreduce.Job: Job job_1504320356950_0011 running in uber mode : false

17/09/02 02:20:51 INFO mapreduce.Job:  map 0% reduce 0%

17/09/02 02:20:58 INFO mapreduce.Job:  map 100% reduce 0%

17/09/02 02:21:05 INFO mapreduce.Job:  map 100% reduce 100%

17/09/02 02:21:06 INFO mapreduce.Job: Job job_1504320356950_0011 completed successfully

17/09/02 02:21:06 INFO mapreduce.Job: Counters: 49

    File System Counters

        FILE: Number of bytes read=118

        FILE: Number of bytes written=241857

        FILE: Number of read operations=0

        FILE: Number of large read operations=0

        FILE: Number of write operations=0

        HDFS: Number of bytes read=174

        HDFS: Number of bytes written=76

        HDFS: Number of read operations=6

        HDFS: Number of large read operations=0

        HDFS: Number of write operations=2

    Job Counters

        Launched map tasks=1

        Launched reduce tasks=1

        Data-local map tasks=1

        Total time spent by all maps in occupied slots (ms)=3828

        Total time spent by all reduces in occupied slots (ms)=4312

        Total time spent by all map tasks (ms)=3828

        Total time spent by all reduce tasks (ms)=4312

        Total vcore-milliseconds taken by all map tasks=3828

        Total vcore-milliseconds taken by all reduce tasks=4312

        Total megabyte-milliseconds taken by all map tasks=3919872

        Total megabyte-milliseconds taken by all reduce tasks=4415488

    Map-Reduce Framework

        Map input records=6

        Map output records=12

        Map output bytes=118

        Map output materialized bytes=118

        Input split bytes=98

        Combine input records=12

        Combine output records=9

        Reduce input groups=9

        Reduce shuffle bytes=118

        Reduce input records=9

        Reduce output records=9

        Spilled Records=18

        Shuffled Maps =1

        Failed Shuffles=0

        Merged Map outputs=1

        GC time elapsed (ms)=186

        CPU time spent (ms)=1200

        Physical memory (bytes) snapshot=297316352

        Virtual memory (bytes) snapshot=4159815680

        Total committed heap usage (bytes)=139595776

    Shuffle Errors

        BAD_ID=0

        CONNECTION=0

        IO_ERROR=0

        WRONG_LENGTH=0

        WRONG_MAP=0

        WRONG_REDUCE=0

    File Input Format Counters

        Bytes Read=76

    File Output Format Counters

        Bytes Written=76

[root@master classes]# /home/hadoop-2.7.4/bin/hadoop fs -ls /

Found 4 items

-rw-r--r--   2 root supergroup         76 2017-09-02 00:57 /hdfs-input.txt

drwxr-xr-x   - root supergroup          0 2017-09-02 02:21 /result-package

drwxr-xr-x   - root supergroup          0 2017-09-02 02:12 /result-self-compile

drwx------   - root supergroup          0 2017-09-02 02:11 /tmp

[root@master classes]#

[root@master classes]#

为啥要删除package，就是因为有包路径的时候调用方式就要 xxx.xxxxx.xxx来执行，而且打包的时候就不能只打class了，目录结构也要一并打进去

同理，自己写的代码也可按照这个方式执行

顺便提一点，如果只是打jar包用

jar -cvf test.jar XXX.class

但是如果要修改MANIFEST.MF，在里面指定mainClass，按照如下方式

#解压文件

jar -xf test.jar 

#在MANIFEST.MF 增加mainclass

Manifest-Version: 1.0

Created-By: 1.6.0_20 (Sun Microsystems Inc.)

Main-class: WordCount

#再打包

jar -cvfm test.jar MANIFEST.MF XXXX.class

这样就可以直接用 java -jar test.jar 运行了，后面不用跟具体的类

Linux巩固记录（5） hadoop 2.7.4下自己编译代码并运行MapReduce程序的更多相关文章

高可用Hadoop平台－运行MapReduce程序
1.概述最近有同学反应,如何在配置了HA的Hadoop平台运行MapReduce程序呢?对于刚步入Hadoop行业的同学,这个疑问却是会存在,其实仔细想想,如果你之前的语言功底不错的,应该会想到自动 ...
hive--构建于hadoop之上、让你像写SQL一样编写MapReduce程序
hive介绍什么是hive? hive:由Facebook开源用于解决海量结构化日志的数据统计 hive是基于hadoop的一个数据仓库工具,可以将结构化的数据映射为数据库的一张表,并提供类SQL查 ...
攻城狮在路上（陆）-- 配置hadoop本地windows运行MapReduce程序环境
本文的目的是实现在windows环境下实现模拟运行Map/Reduce程序.最终实现效果:MapReduce程序不会被提交到实际集群,但是运算结果会写入到集群的HDFS系统中. 一.环境说明: ...
hadoop——在命令行下编译并运行map-reduce程序 2
hadoop map-reduce程序的编译需要依赖hadoop的jar包,我尝试javac编译map-reduce时指定-classpath的包路径,但无奈hadoop的jar分布太散乱,根据自己 ...
Hadoop 系列文章(三) 配置部署启动YARN及在YARN上运行MapReduce程序
这篇文章里我们将用配置 YARN,在 YARN 上运行 MapReduce. 1.修改 yarn-env.sh 环境变量里的 JAVA_HOME 路径 [bamboo@hadoop-senior ha ...
Hadoop YARN上运行MapReduce程序
(1)配置集群 (a)配置hadoop-2.7.2/etc/hadoop/yarn-env.sh 配置一下JAVA_HOME export JAVA_HOME=/home/hadoop/bigdata ...
hadoop 编译代码及运行
搞定了hadoop配置之后,可以写代码运行了,首先要配一下CLASS_PATH,修改/etc/profile export JAVA_HOME=/usr/lib/jvm/java--openjdk-i ...
Hadoop日记Day16---命令行运行MapReduce程序
一.代码编写 1.1 单词统计回顾我们以前单词统计的例子,如代码1.1所示. package counter; import java.net.URI; import org.apache.hado ...
在window下远程虚拟机（centos）hadoop运行mapreduce程序
(注:虽然连接成功但是还是执行不了.以后有时间再解决吧看到的人别参考仅作个人笔记)先mark下 1.首先在window下载好一个eclipse.和拷贝好linux里面hadoop版本对应的插件(我是 ...

随机推荐

Syslog和Windows事件日志收集
Syslog和Windows事件日志收集 EventLog Analyzer从分布式Windows设备收集事件日志,或从分布式Linux和UNIX设备.交换机和路由器(Cisco)收集syslog.事 ...
MySQL查找SQL耗时瓶颈 SHOW profiles
http://blog.csdn.net/k_scott/article/details/8804384 1.首先查看是否开启profiling功能 SHOW VARIABLES LIKE '%pro ...
keras框架的CNN手写数字识别MNIST
参考:林大贵.TensorFlow+Keras深度学习人工智能实践应用[M].北京:清华大学出版社,2018. 首先在命令行中写入 activate tensorflow和jupyter notebo ...
2019.01.20 bzoj5158 Alice&Bob（拓扑排序+贪心）
传送门短代码简单题. 题意简述:对于一个序列XXX,定义其两个伴随序列a,ba,ba,b,aia_iai表示以第iii个数结尾的最长上升子序列长度,bib_ibi表示以第iii个数开头的最长下降 ...
关于Excel分析图插入到论文的问题
为了保证插入到latex图片不失真,可将Excel中的图进行如下操作: 1.将Excel分析图另存为.pdf格式: 2.利用Adobe acrobat裁剪掉空白的部分,另存为.eps格式: 3.将ep ...
C语言三种方法调用数组
#include <stdio.h> /********************************* * 方法1: 第一维的长度可以不指定 * * 但必须指定第二维的长度 * *** ...
UVa 11077 Find the Permutations (计数DP)
题意:给定 n 和 m,问你在 1 ~ n 的所有排列中,有多少个排列满足至少要交换 m 次才能变成 1 2 3 ... n. 析:首先,先考虑一下,某个排列,要变成 1 2 3 .. n,最少要交换 ...
cyclone IV中DDR的一个疑惑
的生成的DDR2 IP中DDR的时钟竟然是双向的,而在arria10中生成的DDR4则是输出,而DDR2的datasheet上也指出ck和ck#是输入,不知为什么? inout mem_clk , i ...
Mysql之数据表操作
数据表操作: 查看当前数据库中所有的表: show tables; 查看当前使用数据库: select database(); 使用数据表: use 表名; 创建数据表: create table 数 ...
Shell编程-09-Shell中的函数
目录基本语法函数执行函数示例函数可以简化程序的代码量,达到更好的代码复用度,因此会让程序变得更加易读.简洁和易修改.其作用就是将需要多次使用的代码整合到一块,使其成为一个整体,然后通过 ...

Linux巩固记录（5） hadoop 2.7.4下自己编译代码并运行MapReduce程序

Linux巩固记录（5） hadoop 2.7.4下自己编译代码并运行MapReduce程序的更多相关文章

随机推荐

热门专题