Hadoop(1)---运行Hadoop自带的wordcount出错问题。

在hadoop2.9.0版本中，对namenode、yarn做了ha，随后在某一台namenode节点上运行自带的wordcount程序出现偶发性的错误（有时成功，有时失败），错误信息如下：

// :: INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
// :: INFO input.FileInputFormat: Total input files to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address
// :: INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1534406793739_0005
// :: INFO impl.YarnClientImpl: Submitted application application_1534406793739_0005
// :: INFO mapreduce.Job: The url to track the job: http://HLJRslog2:8088/proxy/application_1534406793739_0005/
// :: INFO mapreduce.Job: Running job: job_1534406793739_0005
// :: INFO mapreduce.Job: Job job_1534406793739_0005 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1534406793739_0005 failed with state FAILED due to: Application application_1534406793739_0005 failed  times due to AM Container for appattempt_1534406793739_0005_000002 exited with exitCode:
Failing this attempt.Diagnostics: [-- ::48.561]Exception from container-launch.
Container id: container_e27_1534406793739_0005_02_000001
Exit code:
[-- ::48.562]
[-- ::48.574]Container exited with a non-zero exit code . Error file: prelaunch.err.
Last  bytes of prelaunch.err :
Last  bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

[-- ::48.575]
[-- ::48.575]Container exited with a non-zero exit code . Error file: prelaunch.err.
Last  bytes of prelaunch.err :
Last  bytes of stderr :
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

分析与解决：

网上对类似问题解决办法，主要就是添加对应的classpath，测试了一遍，都不好使，说明上诉问题并不是classpath造成的，出错的时候也查看了classpath，都有对应的值，这里贴一下添加classpath的方法。

1、# yarn classpath    注：查看对应的classpath的值

/data1/hadoop/hadoop/etc/hadoop:/data1/hadoop/hadoop/etc/hadoop:/data1/hadoop/hadoop/etc/hadoop:/data1/hadoop/hadoop/share/hadoop/common/lib/*:/data1/hadoop/hadoop/share/hadoop/common/*:/data1/hadoop/hadoop/share/hadoop/hdfs:/data1/hadoop/hadoop/share/hadoop/hdfs/lib/*:/data1/hadoop/hadoop/share/hadoop/hdfs/*:/data1/hadoop/hadoop/share/hadoop/yarn:/data1/hadoop/hadoop/share/hadoop/yarn/lib/*:/data1/hadoop/hadoop/share/hadoop/yarn/*:/data1/hadoop/hadoop/share/hadoop/mapreduce/lib/*:/data1/hadoop/hadoop/share/hadoop/mapreduce/*:/data1/hadoop/hadoop/contrib/capacity-scheduler/*.jar:/data1/hadoop/hadoop/share/hadoop/yarn/*:/data1/hadoop/hadoop/share/hadoop/yarn/lib/*

如果是上述类变量为空，可以通过下面三个步骤添加classpath。

2.修改mapred.site.xml

添加：

<property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

3.yarn.site.xml

添加：

<property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

4.修改环境变量

#vim ~/.bashrc

在文件最后添加下述环境变量：

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

5. source ~/.bashrc

解决报错问题：

从日志可以看出，发现是由于跑AM的container退出了，并没有为任务去RM获取资源，怀疑是AM和RM通信有问题；一台是备RM,一台活动的RM，在yarn内部，当MR去活动的RM为任务获取资源的时候当然没问题，但是去备RM获取时就会出现这个问题了。

修改vim yarn-site.xml

<property>
<!-- 客户端通过该地址向RM提交对应用程序操作 -->
<name>yarn.resourcemanager.address.rm1</name>
<value>master:</value>
</property>
<property>
<!--ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等。 -->
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>master:</value>
</property>
<property>
<!-- RM HTTP访问地址,查看集群信息-->
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>master:</value>
</property>
<property>
<!-- NodeManager通过该地址交换信息 -->
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>master:</value>
</property>
<property>
<!--管理员通过该地址向RM发送管理命令 -->
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>master:</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm1</name>
<value>master:</value>
</property>
<!--
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>slave1:</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>slave1:</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>slave1:</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>slave1:</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>slave1:</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm2</name>
<value>slave1:</value>
</property>
-->

注：标红的地方就是AM向RM申请资源的rpc端口，出错问题就在这里。

红框里面是我在rm1机器（也就是master）上的yarn文件添加的；当然，如果是在slave1里面添加的话就是添加红框上面以.rm1结尾的那几行，其实，说白点，就是要在yarn-site.xml这个配置文件里面添加所有resourcemanager机器的通信主机与端口。然后拷贝到其他机器，重新启动yarn。最后在跑wordcount或者其他程序没在出错。其实这就是由于MR与RM通信的问题，所以在配置yarn-site.xml文件的时候，最好把主备的通信端口都配置到改文件，防止出错。