Hadoop编程踩坑

Hadoop踩坑

在hadoop所有组件编程中，遇到在Windows下运行程序出现

 java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
     at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
     at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
     at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
     at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
     at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
     at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272)
     at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
     at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
     at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
     at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
     at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2214)
     at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2214)
     at scala.Option.getOrElse(Option.scala:121)
     at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2214)
     at org.apache.spark.SparkContext.<init>(SparkContext.scala:322)
     at leftOutJoin.sparkTopN$.main(SparkLefOutJoin.scala:19)
     at leftOutJoin.sparkTopN.main(SparkLefOutJoin.scala)

通常是需要放置一个系统缺失的应用“

null\bin\winutils.exe

”

建议在代码中插入一行

System.setProperty("hadoop.home.dir", "F:\\spack\\hadoop-common-2.2.0-bin-master")

这里我写的路径是我自己的当然，如果你没有

winutils.exe
那你可以下载我上传的：链接