Hadoop调试记录(2)

自从上次调通hbase后很久没有碰hadoop了，今日想写一个mapreduce的小程序。于是先运行了下自带的wordcount示例程序，却报错了。

信息如下：

kevin@ubuntu:~/usr/hadoop/hadoop$ ./bin/hadoop jar hadoop-examples-1.2..jar wordcount readme.txt output
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO util.NativeCodeLoader: Loaded the native-hadoop library
// :: WARN snappy.LoadSnappy: Snappy native library not loaded
// :: INFO mapred.JobClient: Running job: job_201505110806_0003
// :: INFO mapred.JobClient:  map % reduce %
// :: INFO mapred.JobClient: Task Id : attempt_201505110806_0003_m_000002_0, Status : FAILED
Error initializing attempt_201505110806_0003_m_000002_0:
java.io.IOException: Exception reading file:/~/usr/hadoop/hadoop/tmp/mapred/local/ttprivate/taskTracker/kevin/jobcache/job_201505110806_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:)
    at org.apache.hadoop.mapred.TaskTracker$.run(TaskTracker.java:)
    at java.lang.Thread.run(Thread.java:)
Caused by: java.io.FileNotFoundException: File file:/~/usr/hadoop/hadoop/tmp/mapred/local/ttprivate/taskTracker/kevin/jobcache/job_201505110806_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:)
    ...  more

错误是无法读取jobToken的文件，由于程序不会出错，推测是上次在配置hbase时修改hadoop的core-site.xml文件出错了。在stackoverflow上有一个同样的问题，有人建议删除 该配置文件中关于缓存文件目录的配置，因为这一部分是会被系统默认创建的，不需要手动配置。即下面这一部分：

 <!-- In: conf/core-site.xml -->
 <property>
   <name>hadoop.tmp.dir</name>
   <value>/app/hadoop/tmp</value>
   <description>A base for other temporary directories.</description>
 </property> 

删掉这个问题就解决了。关闭hadoop，重启hadoop。出现新问题 JobTracker is in safe mode：

// :: ERROR security.UserGroupInformation: PriviledgedActionException as:kevin cause:org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.SafeModeException: JobTracker is in safe mode
    at org.apache.hadoop.mapred.JobTracker.checkSafeMode(JobTracker.java:)
    at org.apache.hadoop.mapred.JobTracker.getStagingAreaDir(JobTracker.java:)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
    at java.lang.reflect.Method.invoke(Method.java:)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

碰巧stackoverflow上也有相关的问题，应该是权限的问题，运行以下命令即可：

bin/hadoop dfsadmin -safemode leave

问题均解决，wordcount成功运行。