Windows下编译 Hadoop
Windows下编译 Hadoop-2.9.2
系统环境
系统： Windows 10 10.0_x64
maven： Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer： portoc-2.5.0
zlib： 1.2.3-lib
OpenSSL： 1_0_2r
cmake： 3.14.3-win64-x64
Cygwin： 2.897_x86_64
Visual Studio： Visual Studio 2010 Professional
hadoop： hadoop-2.9.2
Hadoop源码包你们的的编译环境要求
Building on Windows
----------------------------------------------------------------------------------
Requirements:
* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')
Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads
If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to
download here:
http://www.microsoft.com/en-us/download/details.aspx?id=8279
The Windows SDK 8.1 is available to download at:
http://msdn.microsoft.com/en-us/windows/bg162891.aspx
Cygwin is neither required nor supported.
编译必须要设置的环境变量
1.JAVA_HOME必须要设置这个环境变量指向jdk的安装目录
2.Platform这个环境变量也必须设置32位系统为
Platform=Win3264位系统为Platform=x643.ZLIB_HOME为你zlib的安装目录例如
ZLIB_HOME=C:\zlib-1.2.7
编译需要注意的点
1.保证你hadoop源码的路径尽量短，比如放在某个盘的根目录
2.保证你的maven仓库的位置尽量短，比如放在某个盘的根目录（注意设置maven的conf文件）
编译过程中可能出现的问题
- 1.Command execution failed. Cannot run program "msbuild" (in directory "C:\hadoop-2.9.2-src\hadoop-common-project\hadoop-common")
解决方法：将C:\Windows\Microsoft.NET\Framework\v4.0.30319放入PATH环境变量
- 2.(compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
解决方法：这个问题是由于Visual Studio没有安装或安装的版本不对，换成对应的版本就好了
- 3.(compile-ms-native-dll) on project hadoop-common: Command execution failed. Process
exited with an error: 1(Exit value: 1) -> [Help 1]
解决方法：这个问题是由于zlib环境的问题，可能缺少ZLIB_HOME或者你下载的zlib的版本不正确或是你的zlib环境变量你们没有包含zlib.h或缺少unistd.h和getopt.h这两个文件可到GitHub上下载这两个文件.
- 4.(dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 2
解决方法：这个问题是由于在编译hadoop-kms这个模块的时候下载Tomcat是由于网络问题出现网络断开连接重新编译也会出现这个问题，可以删除源码重新下载对应的源码进行编译（简单粗暴）
编译
下载hadoop源码选择对应的版本，解压源码后放在某个盘的根目录，进入源码的根目录执行mvn package -Pdist,native-win -DskipTests -Dtar后maven就开始去远程仓库下载需要的依赖这个过程可能需要很久跟带宽速度和墙有很大关系，过程中可能会出现各种问题但是自己都完美解决了，hadoop2.x的版本中我编译了hadoop2.7.7、hadoop-2.8.8、hadoop-2.9.2这三个版本都是编译成功的，但是在编译hadoop3.x版本就会失败，可能3.x对环境的需求相对于2.x有比较大的改变，后续再研究。。。
经过漫长的等待终于出现了久违的画面
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 0.928 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.579 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.780 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.413 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.278 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.491 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.365 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 3.223 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.977 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.304 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:08 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.788 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 9.379 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.052 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 23.895 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 48.088 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 25.830 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 25.398 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 10.032 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.641 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.092 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.053 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.054 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 19.308 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 53.206 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.731 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.056 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 16.246 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 20.362 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.566 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 6.348 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 3.742 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 25.768 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.535 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 6.465 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 3.396 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 3.146 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.784 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 4.994 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 1.584 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.336 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.052 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.535 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.052 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.050 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 7.322 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.128 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 32.088 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 19.070 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.889 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 8.415 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 5.543 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 4.486 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.283 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.521 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.765 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 3.277 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 3.957 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.462 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.628 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 14.629 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.915 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.689 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.238 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.851 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.052 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.016 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 6.852 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.349 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 2.039 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 6.595 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.961 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 7.219 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 3.102 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 5.460 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.486 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.075 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 51.638 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.581 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.055 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:01 min
[INFO] Finished at: 2019-05-08T11:42:52+08:00
[INFO] ------------------------------------------------------------------------
C:\Users\Andy\Downloads\hadoop-2.9.2-src>
大功告成编译后的hadoop放在hadoop-2.9.2-src\hadoop-dist\target目录下。
Windows下编译 Hadoop的更多相关文章
- Windows 10 x64 下编译 Hadoop 源码
Windows 10 x64 下编译 Hadoop 源码 环境准备 Hadoop并没有提供官方的 Windows 10 下的安装包,所以需要自己手动来编译,官方文档中 BUILDING.txt 文件中 ...
- [转] Windows下编译OpenSSL
简述 OpenSSL是一个开源的第三方库,它实现了SSL(Secure SocketLayer)和TLS(Transport Layer Security)协议,被广泛企业应用所采用.对于一般的开发人 ...
- Windows下编译objective-C
Windows下编译objective-C 2011-08-31 14:32 630人阅读 评论(0) 收藏 举报 windowscocoa工具objective clibraryxcode 目录 ...
- 在Windows下编译FFmpeg详细说明
MinGW:一个可自由使用和自由发布的Windows特定头文件和使用GNC工具集导入库的集合,允许你生成本地的Windows程序而不需要第三方C运行时 MinGW,即 Minimalist GNU F ...
- 如何在WINDOWS下编译BOOST C++库 .
如何在WINDOWS下编译BOOST C++库 cheungmine 2008-6-25 写出来,怕自己以后忘记了,也为初学者参考.使用VC8.0和boost1.35.0. 1)下载boost ...
- 在Windows下编译OpenSSL（VS2005和VC6）
需要说明的是请一定安装openssl-0.9.8a . openssl-1.0.0我没有编译成功. 如何在Windows下编译OpenSSL (Vs2005使用Vc8的cl编译器)1.安装Activ ...
- windows下编译java源文件的编码错误
import java.util.Arrays;public class ArrayAsAReference{ public static void main(String[] args) { int ...
- Windows下编译SDL
Windows下编译SDL的理由我就不多说了,无论用VS来编译或调试SDL库都是很方便的.而且SDL源代码中也包含了VC工程,你所要做的只是解压VC工程,进行适当的配置,然后编译.调试. 编译SDL大 ...
- Windows下编译安装 FFmpeg
在Linux/Mac下编译 ffmpeg是非常方便的.但要在 Windows下编译 ffmpeg还真要花点时间.以下就是在 Windowns下编译ffmpeg的步骤: 一.安装Cygwin 在wind ...
随机推荐
- git-scm教程摘要
Git 有三种状态 已提交(committed).已修改(modified)和已暂存(staged) 已提交表示数据已经安全的保存在本地数据库中. 已修改表示修改了文件,但还没保存到数据库中. 已暂存 ...
- typescript装饰器 方法装饰器 方法参数装饰器 装饰器的执行顺序
/* 装饰器:装饰器是一种特殊类型的声明,它能够被附加到类声明,方法,属性或参数上,可以修改类的行为. 通俗的讲装饰器就是一个方法,可以注入到类.方法.属性参数上来扩展类.属性.方法.参数的功能. 常 ...
- 006-多线程-JUC线程池-并发测试程序
一.java代码模拟并发 1.1.一次并发 单次并发测试 1.使用CountDownLatch 等待一个或多个线程一起执行 详细参看:007-多线程-锁-JUC锁-CountDownLatch-闭锁[ ...
- ECMAScript 6复习<一>
1.let和const命令: let不存在变量提升 暂时性死区 let在相同作用域内不允许重复声明 2.块级作用域: 3.全局对象的属性: ; window.a let b = ; window.b ...
- mssql的update :from语法
一条Update更新语句是不能更新多张表的,除非使用触发器隐含更新.而表的更新操作中,在很多情况下需要在表达式中引用要更新的表以外的数据.我们先来讨论根据其他表数据更新你要更新的表 一.MS S ...
- 集合运算 & 聚合函数
SQL 查询之集合运算 & 聚合函数 1.集合运算 1.1.并集运算 UNION 1.2.差集运算 EXCEPT 1.3.交集运算 INTERSECT 1.4.集合运算小结 2.聚合函数 ...
- PHP替代session的方法
PHP替代session的方法 服务器集群的时候 会发现session的问题 一般采用redis 来代替 用账号作为key 因为redis能主从 所以打算用替代session的方法1 cookie代替 ...
- GSVA的使用
GSVA的简介 Gene Set Variation Analysis,被称为基因集变异分析,是一种非参数的无监督分析方法,主要用来评估芯片核转录组的基因集富集结果.主要是通过将基因在不同样品间的表达 ...
- TCP/IP学习笔记10--以太网之基本概念2： 以太网帧格式
"如果错过太阳时你流了泪,那么你也要错过群星了."--- 泰戈尔 前导码(Preamble): 最前面时8个字节的前导码,表示一个以太网帧的开始. 另外,前导码的最后两个比特是&q ...
- Nginx里的root/index/alias/proxy_pass的意思
1.[alias] 别名配置,用于访问文件系统,在匹配到location配置的URL路径后,指向[alias]配置的路径.如: location /test/ { alias /home/sftp/i ...