hadoop执行hbase插入表操作,出错:Stack trace: ExitCodeException exitCode=1:(xjl456852原创)

在执行hbase和mapreduce融合时,将hdfs上的文本文件插入到hbase中,我没有使用"胖包"(胖包就是将项目依赖的jar包放入项目打包后的lib目录中),而是直接将hbase的lib加入到了hadoop的classpath中.

我是将HBase的jar包加到/opt/modules/hadoop/etc/hadoop/hadoop-env.sh中。配置好这个文件,分发到各个节点,改这个配置不用重启集群.

TEMP=`ls /opt/modules/hbase/lib/*.jar`
HBASE_JARS=`echo $TEMP | sed 's/ /:/g'`
HADOOP_CLASSPATH=$HBASE_JARS

可以通过命令查看hadoop的classpath现在包含哪些jar包:

[hadoop@master ~]$ hdfs classpath

　　

运行hadoop的mapreduce程序

hadoop jar xx.jar input '表名'

 

执行后出错:

16/08/11 08:52:05 INFO mapreduce.Job: Running job: job_1470930593079_0001
16/08/11 08:52:24 INFO mapreduce.Job: Job job_1470930593079_0001 running in uber mode : false
16/08/11 08:52:24 INFO mapreduce.Job:  map 0% reduce 0%
16/08/11 08:52:24 INFO mapreduce.Job: Job job_1470930593079_0001 failed with state FAILED due to: Application application_1470930593079_0001 failed 2 times due to AM Container for appattempt_1470930593079_0001_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://master.xjl456852.com:8088/cluster/app/application_1470930593079_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1470930593079_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
	at org.apache.hadoop.util.Shell.run(Shell.java:456)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
16/08/11 08:52:24 INFO mapreduce.Job: Counters: 0

　　

 

网上查询到需要在/opt/modules/hadoop/etc/hadoop/yarn-site.xml中加入mapreduce运行时需要的类库,需要设置yarn.application.classpath:

所以我在yarn-site.xml中加入了如下配置,并加入了hbase的lib目录,配置好这个文件,分发到各个节点,这个配置需要重启集群

<property>
  <name>yarn.application.classpath</name>
      <value>
           /opt/modules/hadoop/etc/*,
           /opt/modules/hadoop/etc/hadoop/*,
           /opt/modules/hadoop/lib/*,
           /opt/modules/hadoop/share/hadoop/common/*,
           /opt/modules/hadoop/share/hadoop/common/lib/*,
           /opt/modules/hadoop/share/hadoop/mapreduce/*,
           /opt/modules/hadoop/share/hadoop/mapreduce/lib/*,
           /opt/modules/hadoop/share/hadoop/hdfs/*,
           /opt/modules/hadoop/share/hadoop/hdfs/lib/*,
           /opt/modules/hadoop/share/hadoop/yarn/*,
           /opt/modules/hadoop/share/hadoop/yarn/lib/*,
		/opt/modules/hbase/lib/*
      </value>
</property>

 

参考:http://blog.chinaunix.net/uid-20776139-id-5586691.html