Hadoop 系列文章(三) 配置部署启动YARN及在YARN上运行MapReduce程序

　　这篇文章里我们将用配置 YARN，在 YARN 上运行 MapReduce。

　　

　　1、修改 yarn-env.sh 环境变量里的 JAVA_HOME 路径

　　[bamboo@hadoop-senior hadoop-2.5.0]$ vim etc/hadoop/yarn-env.sh

　　export JAVA_HOME=/opt/modules/jdk1.7.0_67

 

　　2、修改 etc/hadoop/yarn-site.xml，添加如下的两个属性

 

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-senior.bamboo.com</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>    

 

　　修改 slave 的 hostname(即把 localhost 改为 hostname)

　　vim /etc/hadoop/slaves

　　把 localhost 换为 hostname 

　　hadoop-senior.bamboo.com  

 

　　3、启动 yarn

　　

[bamboo@hadoop-senior hadoop-2.5.0]$ sbin/yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-bamboo-resourcemanager-hadoop-senior.bamboo.com.out

[bamboo@hadoop-senior hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager
starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-bamboo-nodemanager-hadoop-senior.bamboo.com.out

[bamboo@hadoop-senior hadoop-2.5.0]$ jps
10131 NameNode
11931 NodeManager
12033 Jps
10467 DataNode
11679 ResourceManager

 

　　4、修改 mapred-env.sh 和 mapred-site.xml

　　mapred-env.sh (修改 JAVA_HOME 变量地址)

 

　　mapred-site.xml 文件，添加如下的属性

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

 

　　5、在 yarn 上运行 mapreduce

　

运行前需要删除 output 目录，否则运行的时候会报错

[bamboo@hadoop-senior hadoop-2.5.0]$ bin/hdfs dfs -rm -R /user/bamboo/mapreduce/wordcount/output
17/12/31 19:42:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/12/31 19:42:25 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /user/bamboo/mapreduce/wordcount/output

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /user/bamboo/mapreduce/wordcount/input /user/bamboo/mapreduce/wordcount/output

[bamboo@hadoop-senior hadoop-2.5.0]$ bin/hdfs dfs -cat /user/bamboo/mapreduce/wordcount/output/*
17/12/31 19:46:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop 4
hdfs 1
mapreduce 1
nodemanager 1
resourcemanager 1
yarn 2

Ok,YARN 的配置以及启动就到这里。