Hadoop学习历程（四、运行一个真正的MapReduce程序）

上次的程序只是操作文件系统，本次运行一个真正的MapReduce程序。

运行的是官方提供的例子程序wordcount，这个例子类似其他程序的hello world。

1. 首先确认启动的正常：运行 start-all.sh

2. 执行jps命令检查：NameNode，DateNode，SecondaryNameNode，ResourceManager，NodeManager是否已经启动正常。这里我遇到了一个问题，NodeManager没有正常启动。错误信息如下：

2014-01-07 13:46:21,442 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize mapreduce.shuffle
java.lang.IllegalArgumentException: The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:218)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:188)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:338)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386)

经过检查，是配置文件中有点错误，请修改yarn-site.xml文件，更改为如下内容（原因不明）

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

3. 准备数据：在hadoop文件系统中增加input/file1.txt和input/file2.txt

[root@dbserver mapreduce]# hadoop fs -ls /input
Found  items
-rw-r--r--    root supergroup          -- : /input/file1.txt
-rw-r--r--    root supergroup          -- : /input/file2.txt

[root@dbserver mapreduce]# hadoop fs -cat /input/file1.txt
Hello World

[root@dbserver mapreduce]# hadoop fs -cat /input/file2.txt
Hello Hadoop

4. 例子程序的位置在：/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar

hadoop jar ./hadoop-mapreduce-examples-2.2..jar wordcount /input /output

画面输出内容

// :: INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
// :: INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
// :: INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
// :: INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
// :: INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
// :: INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
// :: INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
// :: INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
// :: INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
// :: INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
// :: INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
// :: INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1389074273046_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1389074273046_0001 to ResourceManager at localhost/127.0.0.1:
// :: INFO mapreduce.Job: The url to track the job: http://dbserver:8088/proxy/application_1389074273046_0001/
// :: INFO mapreduce.Job: Running job: job_1389074273046_0001
// :: INFO mapreduce.Job: Job job_1389074273046_0001 running in uber mode : false
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Job job_1389074273046_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
        File System Counters
                FILE: Number of bytes read=
                FILE: Number of bytes written=
                FILE: Number of read operations=
                FILE: Number of large read operations=
                FILE: Number of write operations=
                HDFS: Number of bytes read=
                HDFS: Number of bytes written=
                HDFS: Number of read operations=
                HDFS: Number of large read operations=
                HDFS: Number of write operations=
        Job Counters
                Launched map tasks=
                Launched reduce tasks=
                Data-local map tasks=
                Total time spent by all maps in occupied slots (ms)=
                Total time spent by all reduces in occupied slots (ms)=
        Map-Reduce Framework
                Map input records=
                Map output records=
                Map output bytes=
                Map output materialized bytes=
                Input split bytes=
                Combine input records=
                Combine output records=
                Reduce input groups=
                Reduce shuffle bytes=
                Reduce input records=
                Reduce output records=
                Spilled Records=
                Shuffled Maps =
                Failed Shuffles=
                Merged Map outputs=
                GC time elapsed (ms)=
                CPU time spent (ms)=
                Physical memory (bytes) snapshot=
                Virtual memory (bytes) snapshot=
                Total committed heap usage (bytes)=
        Shuffle Errors
                BAD_ID=
                CONNECTION=
                IO_ERROR=
                WRONG_LENGTH=
                WRONG_MAP=
                WRONG_REDUCE=
        File Input Format Counters
                Bytes Read=
        File Output Format Counters
                Bytes Written=

5. 查看运行结果：

[root@dbserver mapreduce]# hadoop fs -ls /output
Found  items
-rw-r--r--    root supergroup           -- : /output/_SUCCESS
-rw-r--r--    root supergroup          -- : /output/part-r-
[root@dbserver mapreduce]# hadoop fs -cat /output/part-r-
Hadoop
Hello
World