Windows 10 上编译 Hadoop
下载源码
源码下载地址（Source download）：https://hadoop.apache.org/releases.html
这里以 2.9.2 为例，查看源码中的编译说明文件 BUILDING.txt，截取 windows 部分
Requirements: * Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site') Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 The Windows SDK 8.1 is available to download at: http://msdn.microsoft.com/en-us/windows/bg162891.aspx Cygwin is neither required nor supported.
环境准备
1.JDK
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
# 设置系统环境变量
setx /M JAVA_HOME "D:\hadoop\jdk1.8.0_192"
setx /M Path "%Path%;%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin"
2.Maven
http://maven.apache.org/download.cgi
# 设置系统环境变量
setx /M M2_HOME "D:\hadoop\apache-maven-3.6.1"
setx /M Path "%Path%;%M2_HOME%\bin"
conf\settings.xml 仓库配置
<!-- 本地仓库路径 -->
<localRepository>D:\hadoop\repo</localRepository> <!-- 网络仓库地址 -->
<mirrors>
<mirror>
<id>central</id>
<mirrorOf>central</mirrorOf>
<name>aliyunmaven</name>
<url>https://maven.aliyun.com/repository/central</url>
</mirror>
<mirror>
<id>apache.snapshots.https</id>
<mirrorOf>apache.snapshots.https</mirrorOf>
<name>aliyunmaven</name>
<url>https://maven.aliyun.com/repository/apache-snapshots</url>
</mirror>
</mirrors>
3.ProtocolBuffer
https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0
注：除了下载 protobuf 源码外，还需要下载相应版本的编译过的用于 Windows 平台的 protoc 命令（protoc-2.5.0-win32.zip）,该命令用于将 .proto 文件转化为 Java 或 C++ 源文件。
将两个压缩包解压，然后将 protoc.exe 复制到 protobuf-2.5.0\src 目录下
# 安装 ProtocolBuffer
cd D:\hadoop\protobuf-2.5.0\java
mvn test
mvn install # 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\protobuf-2.5.0\src" # 查看版本
protoc --version
4.CMake
# 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\cmake-3.14.3-win64-x64\bin"
5.Visual Studio 2010 Professional
文件名 cn_visual_studio_2010_professional_x86_dvd_532145.iso
SHA1 33D323446131AB9565082D65C9C380BBD7FF228F
文件大小 2.41GB
发布时间 2010-05-26
ed2k://|file|cn_visual_studio_2010_professional_x86_dvd_532145.iso|2591844352|6001253431AFE573E4344F5A0B1D9CAC|/
6.Zlib
使用 MSVC 方式编译，打开开始菜单，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理员身份运行
cd D:\hadoop\zlib-1.2.11
nmake -f win32/Makefile.msc # 编译完成后在该目录下可看到 zlib1.dll 文件
设置系统环境变量
setx /M ZLIB_HOME "D:\hadoop\zlib-1.2.11"
7.Git（需要 bash 命令）
# 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\PortableGit\bin"
编译
1.设置 Platform
setx /M Platform "x64"
2.解压源码，开始编译
打开开始菜单，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理员身份运行
cd D:\hadoop\hadoop-2.9.2-src
mvn package -Pdist,native-win -DskipTests -Dtar # 编译完成后，编译好的文件在目录 hadoop-2.9.2-src\hadoop-dist\target\ 中
编译失败时多编译几次，或自己更换下 Maven 源再试，一些依赖的下载不是很稳定。附上编译成功的日志
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 39.854 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 35.664 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 13.814 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 13.440 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.205 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 28.307 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 26.019 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:47 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:03 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.259 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [10:30 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.421 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:23 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.064 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:11 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:45 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.279 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.455 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.094 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.839 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.846 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.044 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.044 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.375 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [04:12 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.536 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 32.678 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.878 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.167 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 14.856 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.482 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.589 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.674 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.209 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.911 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.163 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:35 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.792 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.213 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.042 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 5.462 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.134 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 23.516 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 13.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.323 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 6.160 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.593 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 2.615 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.280 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.254 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.127 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.367 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 5.503 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.260 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.315 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.355 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.324 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.477 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.219 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.615 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.040 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.513 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:23 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 46.260 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [01:14 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 5.923 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.596 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.338 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [01:43 min]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 30.751 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.557 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.047 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.214 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.554 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.049 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55:18 min
[INFO] Finished at: 2019-04-25T22:47:26+08:00
[INFO] ------------------------------------------------------------------------
Windows 10编译Hadoop 2.6.0源码 - 作业部落 Cmd Markdown 编辑阅读器 （快照）
https://www.cnblogs.com/guoxiaoqian/p/4328812.html
Windows 10 上编译 Hadoop的更多相关文章
- 在Windows 10上安装Oracle 11g数据库出现的问题及解决
在Windows 10上安装Oracle 11g数据库,并且很多次出现过:当安装的进度条进行到快要结束的时候弹出一个提示框.如下: [Java(TM)2 Platform Standard Editi ...
- Windows 10 上，Edge 浏览器不支持插件，因此将不运行 Java
在 Windows 10 上,Edge 浏览器不支持插件,因此将不运行 Java.微软想干嘛?
- DB 查询分析器 6.04 在 Windows 10 上的安装与运行展示
DB查询分析器 6.04 在 Windows 10 上的安装与运行展示 中国本土程序员马根峰(CSDN专访马根峰:海量数据处理与分析大师的中国本土程序员 http://www.csdn.net/art ...
- 如何在Windows 10上运行Docker和Kubernetes？
如何在Windows 10上运行Docker和Kubernetes? 在Windows上学习Docker和Kubernetes,开始的时候会让你觉得无从下手.最起码安装好这些软件都不是一件容易的事情. ...
- Windows 10 上的 Git 如何清除密码？ Git Credential Manager for Windows
Windows 10 上的 Git 如何清除密码? 因为一台新的电脑是 Windows 10 在第一次使用 Git 要求输入密码时把密码给输错了. 之前提交都是说 Token 错了,不再出现提示密码. ...
- Windows 10 上安装 3D Studio Max 2016 报错的解决办法
在 Windows 10 上安装 3D Stuido Max 2016 报错,无法正常安装,查看日志是 VC 运行时安装错误,经过分析发现在 Windows 10 上已经有这些运行时并且版本比安装包中 ...
- 如何在 Windows 10 上安装 WSL 2
翻译自 Joey Sneddon 2020年10月30日的文章<How to Install WSL 2 on Windows 10> [1] 如果您想在最新的 Windows 版本中尝试 ...
- windows 10 上源码编译OpenCV并支持CUDA | compile opencv with CUDA support on windows 10
本文首发于个人博客https://kezunlin.me/post/6580691f/,欢迎阅读! compile opencv with CUDA support on windows 10 Ser ...
- windows 10上源码编译dlib教程 | compile dlib on windows 10
本文首发于个人博客https://kezunlin.me/post/654a6d04/,欢迎阅读! compile dlib on windows 10 Series Part 1: compile ...
随机推荐
- 中国移动能力开放商店OneNET View数据可视化公测 10分钟轻便生成行业可视化界面
随着云计算,5G技术,人工智能等底层技术的发展,万物互联时代已经到来,同时带来了海量数据,如何效果好.低成本.短时间的表现据,成为物联网行业从业者和公司的当务之急. OneNET View传统的数据展 ...
- iis问题
1.8.5版本的iis配置完成,不能打开网页显示503.原因是应用程序池,加载用户配置文件失败,然后应用程序池关闭了.关闭它就可以正常打开了. server2008,打不开网页,打开url显示目录的原 ...
- redis对sorted_set进行的相关操作
redis对sorted_set(有序集合)类型操作的相关命令以及如何在python使用这些命令 redis对sorted_set(有序集合)类型操作的命令: 命令 语法 概述 返回值 Redis Z ...
- java 图片处理 base64编码和图片二进制编码相互转换
今天在弄小程序项目时,涉及上传图片的更改. 以下是代码: /** * -> base64 * @param imgFile * @return * @throws IOException */ ...
- 使用栈实现队列(2)(Java)
class MyQueue { private Stack s1; private Stack s2; public MyQueue(int size) { this.s1 = new Stack(s ...
- Node.js中的console.log()输出彩色字体
转自:https://www.jianshu.com/p/cca3e72c3ba7 console.log('\033[42;30m DONE \033[40;32m Compiled success ...
- js 简单弹框toast
新建toast.js文件 function Toast(msg,duration){ duration=isNaN(duration)?3000:duration; var m = document. ...
- MySQL--7种join连接
一,定义: 1)LEFT JOIN / LEFT OUTER JOIN:左外连接 左向外连接的结果集包括:LEFT OUTER子句中指定的左表的所有行,而不仅仅是连接列所匹配的行.如果左表的某行在右表 ...
- Laravel-任务调度
简介Cron 是 UNIX.SOLARIS.LINUX 下的一个十分有用的工具,通过 Cron 脚本能使计划任务定期地在系统后台自动运行.这种计划任务在 UNIX.SOLARIS.LINUX下术语为 ...
- vue-cli3相关
此时做的一个vue-cli3项目build后,app.js达到了10M,主要为elementui.quill等组件: 最开始使用“compression-webpack-plugin”插件根据网上的说 ...