Hadoop 3.2.1 win10 64位系统 vs2015 编译

Hadoop 3.2.1 win10 64位系统 vs2015 编译

1        环境配置

1.1   JDK下载安装

1.1.1         下载

JDK 1.8    （jdk1.8.0_102）
   下载地址    ： http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html（按照电脑版本下载安装）

1.1.2        
安装

解压到指定文件夹

（1） 安装JDK 
（2）新建系统变量JAVA_HOME=D:\Program
Files\Java\jdk1.8.0_102 
（3）编辑系统变量Path，新增%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin

1.2  
Marven下载和安装

1.2.1        
下载

https://blog.csdn.net/changge458/article/details/53576178

从该网站 http://maven.apache.org/download.cgi 下载

1.2.2        
安装

解压到文件夹D:\marven\apache-maven-3.6.3

添加系统环境变量：MARVEN_HOME

D:\marven\apache-maven-3.6.3\

在系统环境变量path中加入

测试mvn是否安装成功，打开cmd，以管理员身份运行，否则容易报错：'mvn' 不是内部或外部命令，也不是可运行的程序

查看是否配置成功可在黑窗口中输入 mvn –v 出现如下图所示说明配置成功

打开D:\marven\apache-maven-3.6.3\conf\settings.xml
加入阿里云镜像，找到mirrors标签，添加如下阿里云镜像地址，marven下载库从这里下载更快速。

<mirror>

<id>nexus-aliyun</id>

<mirrorOf>central</mirrorOf>

<name>Nexus aliyun</name>

<url>http://maven.aliyun.com/nexus/content/groups/public</url>

</mirror>

1.3  
编译安装protobuff

Protobuff是用于序列化数据的，hadoop编译需要依赖这个库。

1.3.1        
下载

ProtocolBuffer
2.5.0    （两个文件protobuf-2.5.0.zip  protoc-2.5.0-win32.zip
）

下载地址    ：
https://github.com/google/protobuf/releases/tag/v2.5.0

注：除了下载protobuf源码外，还需要下载相应版本的编译过的用于Windows平台的protoc命令（protoc-2.5.0-win32.zip）,该命令用于将.proto文件转化为Java或C++源文件。

1.3.2        
安装

①
解压ProtocolBuffer到指定目录

②
解压protoc-2.5.0-win32.zip，将protoc.exe复制到C:\WorkSpace\protobuf-2.5.0\src目录下

③
安装ProtocolBuffer，打开CMD命令行

cd
C:\WorkSpace\protobuf-2.5.0\java

mvn
test

mvn
install

protoc
–version    这个命令失败

将protoc.exe所在路径C:\WorkSpace\protobuf-2.5.0\src，添加到系统变量Path

1.4  
Git下载和安装

Git是在window的cmd黑窗口中，可以使用linux的命令。不安装会出现错误

Failed
to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on
project hadoop-project-dist: Command execution failed.: Cannot run program
"bash" (in directory
"D:\hadoop\hadoop-3.2.1-src\hadoop-project-dist\target"):
CreateProcess error=2, 系统找不到指定的文件。 -> [Help 1]

1.4.1        
下载

https://git-for-windows.github.io/

1.4.2        
安装

（1）

（2）、
      

（3）、
      

（4）、
      

（5）、

1.5  
CMake 安装

1.5.1        
下载

https://cmake.org/download/

Windows
win64-x64 ZIP  cmake-3.16.0-win64-x64.zip

1.5.2        
安装

解压到指定文件夹即可，path环境变量添加bin路径。

D:\hadoop\cmake-3.16.0-win64-x64\bin

1.6  
Zlib下载安装

1.6.1        
下载

http://jaist.dl.sourceforge.net/project/libpng/zlib/1.2.8/

、

1.6.2        
安装

解压到指定文件夹，添加zlib系统变量

2       
编译hadoop

2.1  
升级项目Visual Studio的版本

编译hadoop时，自动编译会去根据项目文件找到编译器visual studio去编译，hadoop3.2.1默认是VS2010，现在要用VS2015去编译，需要将项目文件winutils.sln和native.sln升级为vs2015。由于Visual
Studio版本问题，官方默认使用Visual Studio 2010 Professional，但本文采用Visual
Studio 2015，因此对于生成失败的项目，需要用Visual Studio 2015重新打开，会弹出升级的对话框，升级项目至Visual Studio 2015版本即可：

(1）      
Window工具编译

D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\winutils

（2）hadoop.dll编译

D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\native

这个项目编译输出的就是hadoop.dll文件，编辑编译生成按钮，会报很多头文件错误，是因为投啊文件件包含了..\..\..\target\native\javah，这个文件夹内的头文件，编译hadoop时，编译命令自动先复制头文件到此文件夹，然后才能编译。

打开cmd窗口，进入路径D:\hadoop\hadoop-3.2.1-src\，执行下面的命令：

mvn package -Pdist,native-win-DskipTests -Dtar

2.2  
Apache
Hadoop Common编译错误

2.2.1        
convert-ms-winutils错误描述

错误描述

[INFO]
Apache Hadoop Auth Examples ........................ SUCCESS [ 12.916 s]

[INFO]
Apache Hadoop Common ............................... FAILURE [02:06 min]

[INFO]
Apache Hadoop NFS .................................. SKIPPED

[INFO]
Apache Hadoop KMS .................................. SKIPPED

[INFO]
------------------------------------------------------------------------

[ERROR]
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(convert-ms-winutils) on project hadoop-common: Command execution failed.:
Process exited with an error: 1 (Exit value: 1) -> [Help 1]

[ERROR]

[ERROR]
To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR]
Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR]
For more information about the errors and possible solutions, please read the
following articles:

[ERROR]
[Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR]
After correcting the problems, you can resume the build with the command

[ERROR]   mvn <args> -rf :hadoop-common

解决办法：

（1）   确认2.1中两个项目都升级成功，sln文件都是显示Visual
studio 14

（2）   VS 2015 x64 native Tools 工具执行命令mvn
package -Pdist,native-win -DskipTests -Dtar -e -X

2.2.2        
Javah错误

错误描述

[ERROR]
Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah
(default) on project hadoop-common: Error running javah command: Error
executing command line. Exit code:1 -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah
(default) on project hadoop-common: Error running javah command

解决办法

因为D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common路径下的pom.xml文件中，<javahPath>${env.JAVA_HOME}/bin/javah</javahPath>采用env.JAVA_HOME
无法识别，所以讲pom.xml文件中换成javah的绝对路径：D:\Java\jdk1.8.0_181\bin\,一共有两个地方。

然后命令加入参数执行–rf :hadoop-common 表示从hadoop-common开始编译：

mvn
package -Pdist,native-win-DskipTests –Dtar –rf :hadoop-common

2.3 
HDFS Native Client编译错误

2.3.1        
错误描述

[INFO]
Apache Hadoop HDFS Native Client ................... FAILURE [02:26 min]

[ERROR]
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client: An Ant BuildException has occured:
exec returned: 1

[ERROR]
around Ant part ...<exec failonerror="true"
dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
executable="cmake">... @ 5:140 in
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

[ERROR]
-> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client: An Ant BuildException has occured:
exec returned: 1

around
Ant part ...<exec failonerror="true"
dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
executable="cmake">... @ 5:140 in
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

2.3.2        
解决方法

D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml文件打开，修改如下部分的true为false；

2.4 
hadoop-hdfs-native-client :RelWithDebInfo编译错误

2.4.1        
错误描述

Finished
at: 2019-12-01T18:20:30+08:00

[INFO]
------------------------------------------------------------------------

[ERROR]
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client: An Ant BuildException has occured:
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo
does not exist.

[ERROR]
around Ant part ...<copy
todir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">...
@ 13:101 in D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

[ERROR]
-> [Help 1]

2.4.2        
解决方法

D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo
does not exist.错误是这个目录不存在，则创建这个目录

2.5 
执行maven-plugin:1.3.1:exec
(pre-dist)失败

2.5.1        
错误描述

Failed
to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on
project hadoop-hdfs-native-client: Command execution failed.: Process exited
with an error: 1 (Exit value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(pre-dist) on project hadoop-hdfs-native-client: Command execution failed.

2.5.2        
解决办法

下载安装Cygwin
下载：http://www.cygwin.com/setup-x86_64.exe ,安装
配置D:\cygwin64\bin到PATH

然后打开cygwin64的terminal窗口，执行命令mvn package -Pdist,native-win
-DskipTests -Dtar -e -X -rf :hadoop-hdfs-native-client

2.6 
Msbuild编译错误

2.6.1        
错误描述

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client:
An Ant BuildException has occured: Execute failed:
java.io.IOException: Cannot run program "msbuild" (in directory
"D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native"):
CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒▒▒▒▒ļ▒▒▒

[ERROR] around Ant part ...<exec failonerror="false"
dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
executable="msbuild">... @ 9:143 in
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

2.6.2        
解决办法

下载安装vscode  https://code.visualstudio.com/docs/?dv=win

将路径C:\Program Files
(x86)\MSBuild\14.0\Bin添加到path环境变量。

2.7 
maven-surefire-plugin编译错误

2.7.1        
错误描述

Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test
(default-test)

2.7.2        
解决方法

在D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\pom.xml找到org.apache.maven.plugins，在configuration中添加<testFailureIgnore>true</testFailureIgnore>屏蔽测试错误。

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-surefire-plugin</artifactId>

<configuration>

<testFailureIgnore>true</testFailureIgnore>

<forkCount>${testsThreadCount}</forkCount>

<reuseForks>false</reuseForks>

<argLine>${maven-surefire-plugin.argLine}
-DminiClusterDedicatedDirs=true</argLine>

<systemPropertyVariables>

<testsThreadCount>${testsThreadCount}</testsThreadCount>

<test.build.data>${test.build.data}/${surefire.forkNumber}</test.build.data>

<test.build.dir>${test.build.dir}/${surefire.forkNumber}</test.build.dir>

<hadoop.tmp.dir>${hadoop.tmp.dir}/${surefire.forkNumber}</hadoop.tmp.dir>

<!-- Due to a Maven quirk,
setting this to just -->

<!-- surefire.forkNumber
won't do the parameter substitution. -->

<!-- Putting a prefix in
front of it like "fork-" makes it -->

<!-- work. -->

<test.unique.fork.id>fork-${surefire.forkNumber}</test.unique.fork.id>

</systemPropertyVariables>

</configuration>

</plugin>

2.8 
Apache Hadoop Distribution编译错误

2.8.1        
错误描述

[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (dist) on project hadoop-dist:
Command execution failed.: Process exited with an error: 1 (Exit value: 1)
-> [Help 1]

2.8.2        
解决办法

执行命令mvn package -Pdist,native-win
-DskipTests -Dtar -e -X -rf :hadoop-dist，加上-e和-X参数显示详细的错误。会发现是目录结构问题，找不到如下路径。解决方法就是创建缺少的路径，并将生成的jar包成果物复制到路径下

[DEBUG] Executing command line: [bash,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target]

cp: cannot stat
'/d/hadoop/hadoop-3.2.1-src/hadoop-common-project/hadoop-kms/target/hadoop-kms-3.2.1/*': No such file or
directory

2.9 
hadoop-dist: exec (toolshooks)编译错误

2.9.1        
错误描述

Executing command line: [bash,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../hadoop-tools]

找不到文件 - *.tools-builtin.txt

D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker:
line 137:
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new:
No such file or directory

mv: cannot stat
'D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new':
No such file or directory

Rewriting
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh

[INFO]
------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop
Distribution 3.2.1:

[INFO]

[INFO] Apache Hadoop Distribution
......................... FAILURE [ 25.920 s]

[INFO] Apache Hadoop Client Modules
....................... SKIPPED

[INFO] Apache Hadoop Cloud Storage
........................ SKIPPED

[INFO] Apache Hadoop Cloud Storage Project
................ SKIPPED

[INFO]
------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO]
------------------------------------------------------------------------

[INFO] Total time:  33.530 s

[INFO] Finished at:
2019-12-08T11:56:12+08:00

[INFO]
------------------------------------------------------------------------

[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (toolshooks) on project
hadoop-dist: Command execution failed.: Process exited with an error: 1 (Exit
value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(toolshooks) on project hadoop-dist: Command execution failed.

2.9.2        
解决方法

（1）错误原因和2.8相似，缺少了文件夹和文件，按照错误提示创建文件夹，找到缺少的文件放入指定文件夹。

D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../hadoop-tools]

找不到文件 - *.tools-builtin.txt

D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker:
line 137:
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new:
No such file or directory

mv: cannot stat
'D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new':
No such file or directory

Rewriting
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh

（2）根据描述是添加了/etc/hadoop文件夹，并且从D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\conf
\ 中追到了hadoop-env.sh复制过去，但是发现在执行编译时，总是会自动删除这个文件夹和文件。在执行下面的命令时，删除的：

[DEBUG] Executing command line: [bash,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target]

Current directory /d/hadoop/hadoop-3.2.1-src/hadoop-dist/target

$ rm -rf hadoop-3.2.1

$ mkdir hadoop-3.2.1

$ cd hadoop-3.2.1

（3）打开D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching文件，发现dist-layout-stitching中有一个语句run rm
-rf "hadoop-${VERSION}"会删除hadoop-3.2.1文件夹，然后再新建，新建之后再复制，所以怎么加etc/hadoop文件夹都会被删除

（4）在文件末尾处添加如下代码，用代码去创建文件夹，并复制hadoop-env.sh文件。

run mkdir "etc"

run cd "etc"

run mkdir "hadoop"

run cd "hadoop"

run copy 
"${ROOT}\hadoop-common-project\hadoop-common\src\main\conf\hadoop-env.sh"

（5）执行命令>mvn package -Pdist,native-win -DskipTests -Dtar -e -X -rf
:hadoop-dist

3      
大工告成

[DEBUG]  
(f) siteDirectory =
D:\hadoop\hadoop-3.2.1-src\hadoop-cloud-storage-project\src\site

[DEBUG]  
(f) skip = false

[DEBUG] -- end configuration --

[INFO] No site descriptor found: nothing to
attach.

[INFO]
------------------------------------------------------------------------

[INFO] Reactor Summary for Apache Hadoop
Distribution 3.2.1:

[INFO]

[INFO] Apache Hadoop Distribution
......................... SUCCESS [ 39.961 s]

[INFO] Apache Hadoop Client Modules
....................... SUCCESS [  5.721
s]

[INFO] Apache Hadoop Cloud Storage
........................ SUCCESS [  10:36
h]

[INFO] Apache Hadoop Cloud Storage Project
................ SUCCESS [  1.471 s]

[INFO]
------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO]
------------------------------------------------------------------------

[INFO] Total time:  10:37 h

[INFO] Finished at:
2019-12-09T10:05:07+08:00

[INFO]
------------------------------------------------------------------------

自己开发了一个股票智能分析软件，功能很强大，需要的点击下面的链接获取：

https://www.cnblogs.com/bclshuai/p/11380657.html

参考文献

https://blog.csdn.net/qq_37475168/article/details/90746823