[b0009] 玩Hadoop中碰到的各种错误

1. Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found

hadoop@ssmaster:~/java_program$ hadoop  jar hadoop_mapr_wordcount.jar  /input /output
// :: INFO client.RMProxy: Connecting to ResourceManager at ssmaster/192.168.249.144:
// :: WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
// :: WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1477315002921_0003
// :: INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
// :: INFO impl.YarnClientImpl: Submitted application application_1477315002921_0003
// :: INFO mapreduce.Job: The url to track the job: http://ssmaster:8088/proxy/application_1477315002921_0003/
// :: INFO mapreduce.Job: Running job: job_1477315002921_0003
// :: INFO mapreduce.Job: Job job_1477315002921_0003 running in uber mode : false
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
    at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:)
    ...  more

// :: INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
    at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:)
    ...  more

// :: INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:)
    at org.apache.hadoop.mapred.YarnChild$.run(YarnChild.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:)

背景：Hadoop2.6.4 Ubuntu， hadoop jar xxx.jar  运行程序时报错

原因：在代码中没有指定执行类

解决：

添加 job.setJarByClass(WordCount2.class);

解决

2  Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://ssmaster:9000/data/paper.txt, expected: file:///

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://ssmaster:9000/data/paper.txt, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:)
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
    at hadoop.hdfs.Hdfs_Download.main(Hdfs_Download.java:)

背景：Hadoop2.6.4 Ubuntu, windows  eclipse 已经配置好win下运行hadoop插件环境，参考[] ，运行hdfs时报错 (下载一个文件到windows本地)

       路径设置如下 src = new Path("hdfs://ssmaster:9000/data/paper.txt" );

       out = new FileOutputStream ("c:\\paper.txt");

解决：

     将 core-site.xml，hdfs-site.xml，mapred-site.xml 复制到eclipse工程的src目录下

解决

3  log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

背景： Hadoop2.6.4 Ubuntu, windows  eclipse 运行程序

解决： 工程下的src目录新建一个文件 log4j.properties，添加内容（网上找)

解决

4 main"java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

背景：window 7 64，eclipse 执行 mapreduce wordcount 程序

原因：win 下开发的问题

解决：

源码包 hadoop-2.6.4-src.tar.gz解压，hadoop-2.6.4-src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio下NativeIO.java 复制到对应的Eclipse的project

修改如下地方

如果没有解决，按照 Y.2.1 中配置

[0008]  http://www.cnblogs.com/sunzebo/p/5991563.html