Mac 下编译 Hadoop
Mac 下编译 Hadoop-2.9.2
系统环境
系统: Mac OS_10.14.4
maven: Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer: portoc-2.5.0
zlib: 1.2.3-lib
OpenSSL: 1_0_2r
cmale: 3.14.3-win64-x64
Cygwin: 2.897_x86_64
hadoop: hadoop-2.9.2
snappy:
搭建编译环境
1.java环境在oracle官网下载安装安装好配置好环境变量
2.maven我用的是brew直接安装的`brew install maven`
3.cmake 使用brew安装 `brew install cmake`
4.openssl 使用brew安装 `brew install openssl`
5.snappy 使用brew安装 `brew install snappy`
6.zlib 使用brew 安装 `brew install zlib`
7.protobuf由于必须使用2.5.0版本需要自行[下载](https://github.com/protocolbuffers/protobuf/releases)安装
下载后解压后进入protobuf目录
# 配置
$ ./configure --prefix=/usr/local/Cellow/protoc/2.5/
# 编译
$ make
# 安装
$ make install
# 配置环境变量 `~/.bash_profile` 或 `~/.zshrc`
export PATH="/usr/local/Cellow/protoc/2.5/bin:$PATH"
# 验证protoc版本
$ protoc --version
过程中可能遇到的问题
- 1.[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...... @ 5:153 in /Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
解决办法: 由于这个xml中的编译指令中需要用到一个环境变量OPENSSL_ROOT_DIR
由于我们前面没有配置所以他在执行的时候就会报错,需要在~/.bash_profile
或 ~/.zshrc
中指定OPENSSL_ROOT_DIR
的安装目录我的配置是export OPENSSL_ROOT_DIR="/usr/local/Cellor/openssl/1.0.2r"
- 2.由于OpenSSL爆出有重大的漏洞Mac OS没有使用OpenSSL而是使用了
LibreSSl
可以在你的Mac下执行which openssl
然后找到/usr/bin/openssl
执行/usr/bin/openssl version
就会出现LibreSSl 2.6.4,但是编译hadoop必须要使用openssl,可以吧openssl的引用改一个名称,但是Mac是不允许你关闭Mac的保护模式但是不建议这样做,我们可以修改系统的shell的环境变量使自己配置的环境变量覆盖系统的环境变量,例如:export PATH="/usr/local/Cellor/openssl/1.0.2r/bin:$PATH"
编译hadoop
# 下载[hadoop源码](https://hadoop.apache.org/releases.html)
# 解压缩后进入hadoop源码路径
# 执行一条maven命令(你的maven需要配置到环境变量中)
$ mvn package -Pdist,native -DskipTests -Dtar
# 编译过程不会太长,但是下载各种依赖就会浪费很长时间,耐心等待!
# 过程中可能会出现各各样的问题我只是列举了我花费比较久时间解决的问题,如果无法解决可以留言。
编译完成
经过漫长的编译终于出现了令人兴奋的画面
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.679 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.158 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.071 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.897 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.297 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.699 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.280 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 2.940 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.245 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.548 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.707 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 9.596 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.046 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 18.270 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 46.683 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 7.154 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.156 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 3.215 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.857 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.849 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.100 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.073 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 10.318 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 23.624 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.413 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.091 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 8.550 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 26.322 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.491 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 4.873 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 3.360 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 16.652 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.993 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 4.276 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.371 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 2.464 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.747 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 5.310 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 1.894 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.053 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.099 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.366 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.061 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.098 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 7.639 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.157 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 15.248 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 9.780 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.561 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 8.028 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.123 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 4.353 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.349 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 3.550 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.609 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 2.987 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 3.530 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.456 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.654 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.363 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.833 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.550 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.378 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.101 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 6.900 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.804 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 7.562 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.022 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 9.670 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 5.619 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.638 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.075 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 10.540 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 12.875 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.670 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.058 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:02 min]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.473 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.042 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:17 min
[INFO] Finished at: 2019-05-06T05:03:29-07:00
[INFO] ------------------------------------------------------------------------
MacBookPro:hadoop-2.9.2-src leone$
大功告成编译后的hadoop放在hadoop-2.9.2-src\hadoop-dist\target
目录下。
Mac 下编译 Hadoop的更多相关文章
- Windows 10 x64 下编译 Hadoop 源码
Windows 10 x64 下编译 Hadoop 源码 环境准备 Hadoop并没有提供官方的 Windows 10 下的安装包,所以需要自己手动来编译,官方文档中 BUILDING.txt 文件中 ...
- MAC 下编译 ANDROID P 源码 提示 internal error: Could not find a supported mac sdk: ["10.10" "10.11" "10.12" "10.13"]
MAC 下编译 ANDROID P 源码出现下面的问题: ninja: no work to do. [21/21] out/soong/.bootstrap/bin/soong_build out/ ...
- mac下编译FFmpeg-Android
参考: [史上最傻瓜的]mac下编译FFmpeg-Android http://blog.csdn.net/ashqal/article/details/9381037
- 64位centos 下编译 hadoop 2.6.0 源码
64位os下为啥要编译hadoop就不解释了,百度一下就能知道原因,下面是步骤: 前提:编译源码所在的机器,必须能上网,否则建议不要尝试了 一. 下载必要的组件 a) 下载hadoop源码 (当前最新 ...
- eclipse下编译hadoop源代码(转)
hadoop是一个分布式存储和分布式计算的框架.在日常使用hadoop时,我们会发现hadoop不能完全满足我们的需要,我们可能需要修改hadoop的源代码并重新编译.打包. 下面将详细描述如何从sv ...
- 在windows环境下编译hadoop
1.环境准备 1.1 JDK的安装 下载jdk1.6.0_43(这里务必要使用jdk的1.6版本,因为使用JDK1.7版本编译hadoop的时候,很多Maven依赖下载不完整,最终会报错)解压到,并将 ...
- mac下编译安装grafana 4.2.0
go语言在开发效率和运行效率中的优势让很多人青睐,所以有倾向打算转向go语言的开发. 下面介绍在Mac OS X中golang的开发环境配置. 1.安装brew brew是一个mac下的由ruby开发 ...
- Mac下编译libpomelo静态库,并在cocos2dx项目中引用
最近在学习cocos2dx的过程中需要和服务器进行交互,所以这几天在学习libpomelo静态库的编译和使用.之前在windows系统下编译libpomelo,并在VS中引入比较顺利:但是,目前对Ma ...
- Windows下编译 Hadoop
Windows下编译 Hadoop-2.9.2 系统环境 系统: Windows 10 10.0_x64 maven: Apache Maven 3.6.0 jdk: jdk_1.8.0_201 Pr ...
随机推荐
- APP:目录
ylbtech-APP:目录 1.返回顶部 2.返回顶部 3.返回顶部 4.返回顶部 5.返回顶部 6.返回顶部 7.返回顶部 8.返回顶部 9.返回顶部 ...
- vue plupload 的使用
1. 首选npm安装plupload 2. 阿里云OSS PHP 安全上传 <template> <div class="imgUpload"> aaa ...
- 记一次haproxy反向代理配置
首先借用一下前辈的话,解释下反向代理是什么? 反向代理:以代理服务器来接受internet上的连接请求,然后将请求转发给内部网络上的服务器,并将从服务器上得到的结果返回给internet上请求连接的客 ...
- 【FreeMarker】FreeMarker使用(三)
搭建一个 1.FreeMarker取值 <!DOCTYPE html> <html> <head> <meta charset="UTF-8&quo ...
- [译]使用Pandas读取大型Excel文件
上周我参加了dataisbeautiful subreddit上的Dataviz Battle,我们不得不从TSA声明数据集创建可视化.我喜欢这种比赛,因为大多数时候你最终都会学习很多有用的东西. 这 ...
- 无法嵌入互操作类型"NationalInstruments.TestStand.Interop.UI.ExecutionViewOptions"。请改用适用的接口
参考一下文章说明, 修改Interop.UI动态库的引入属性为 False,不再报错: VS2010,VS2012,VS2013中,无法嵌入互操作类型“……”,请改用适用的接口的解决方法 在VS2 ...
- Apache使用内置插件mod_php解析php的配置
apache安装完毕之后,修改httpd.conf配置文件,添加代码如下: LoadModule php5_module modules/libphp5.so <FilesMatch \.php ...
- JAVA协程 纤程 与Quasar 框架
ava使用的是系统级线程,也就是说,每次调用new Thread(....).run(),都会在系统层面建立一个新的线程,然鹅新建线程的开销是很大的(每个线程默认情况下会占用1MB的内存空间,当然你愿 ...
- Linux 指令表
Linux简介(了解) Linux介绍 Linux是类UNIX计算机的统称 Linux操作系统的内核也是Linux Linux是由芬兰大学生Linux Torvalds于1991年编写的 Linux这 ...
- Andrew Ng机器学习课程10
Andrew Ng机器学习课程10 a example 如果hypothesis set中的hypothesis是由d个real number决定的,那么用64位的计算机数据表示的话,那么模型的个数一 ...