Ubuntu下编译 Hadoop2.9
Ubuntu 下编译 Hadoop-2.9.2
系统环境
系统： ubuntu-18.10-desktop-amd64
maven： Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer： portoc-2.5.0
OpenSSL： OpenSSL 1.1.1
cmake： cmake3.12.1
hadoop： hadoop-2.9.2
安装所需环境
# 安装各种所需依赖库
$ sudo apt-get install autoconf automake libtool zlib1g-dev pkg-config libssl-dev libsnappy-dev libbz2-dev
# 安装ssh-server
$ sudo apt-get install ssh-server
# 安装openssl
$ sudo apt-get install openssl
# 安装cmake
$ sudo apt-get install cmake
# 安装编译c++库
$ sudo apt-get install g++
# 安装protobuf
# 下载protobuf-2.5.0
$ wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
# 解压缩
$ tar -zxvf protobuf-2.5.0.tar.gz
# 进入protobuf解压缩目录配置
$ ./configur --prefix=/usr/local/protoc-2.5.0
# 编译安装
$ make && make install
# 配置到环境变量中~/.bash_profile
export PATH="/usr/local/protoc-2.5.0/bin:$PATH"
编译过程中可能遇到的问题
注意 libssl-dev安装换成 libssl1.0-dev版本。 我之前就是默认安转，在编译OpenSSL功能的时候一直卡主编译不通过。具体错误形式如下
[exec] make[2]: *** [CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o] Error 1
[exec] make[1]: *** [CMakeFiles/hadoop_static.dir/all] Error 2
[exec] make: *** [all] Error 2
# 移除libssl-dev
$ sudo apt-get remote libssl-dev
# 安装libssl1.0-dev
$ sudo apt-get install libssl1.0-dev
开始编译 hadoop
$ mvn package -Pdist,native -DskipTests -Dtar
命令执行后需要等待他下载编译所需的所有依赖过程可能会很久
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 0.758 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.426 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.470 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.169 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.134 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.145 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.224 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.923 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 2.653 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.823 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 40.786 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.925 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 8.053 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.050 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 46.997 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 42.883 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 5.907 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 11.476 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 5.320 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.055 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 11.390 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.053 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.620 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 25.431 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.577 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.067 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 6.540 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 25.577 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.779 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 11.509 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.671 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 12.742 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.760 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.285 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.976 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.730 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 2.516 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 23.469 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 1.727 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.037 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.611 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.039 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.041 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.027 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 4.166 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.126 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 11.880 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 8.171 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 1.818 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 4.549 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.999 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 3.505 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.104 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.656 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.052 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 9.580 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.545 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.964 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.355 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 2.651 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.451 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.281 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 0.910 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.269 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 4.864 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.166 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.425 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 3.589 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.737 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 4.883 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.535 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.363 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 2.404 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.732 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 17.061 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.035 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 42.773 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 2.183 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.229 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:19 min
[INFO] Finished at: 2019-05-08T17:23:30+08:00
[INFO] ------------------------------------------------------------------------
leone@leone-ubuntu:~/Downloads/hadoop-2.9.2-src$
大功告成编译后的hadoop放在hadoop-2.9.2-src\hadoop-dist\target目录下。
Ubuntu下编译 Hadoop2.9的更多相关文章
- 在Ubuntu下编译hadoop2.5.x
在Ubuntu下编译hadoop2.5.x 参考博客:http://www.aboutyun.com/thread-8130-1-1.html 1 下载hadoop源码: (1) http://www ...
- Ubuntu下用hadoop2.4搭建集群（伪分布式）
要真正的学习hadoop,就必需要使用集群,可是对于普通开发人员来说,没有大规模的集群用来測试,所以仅仅能使用伪分布式了.以下介绍怎样搭建一个伪分布式集群. 为了节省时间和篇幅,前面一些步骤不再叙述. ...
- ubuntu下编译caffe
Ubuntu下编译caffe 纯粹是个人编译的记录.不用CUDA(笔记本是amd卡,万恶的nvidia):不手动编译依赖包(apt-get是用来干啥的?用来直接装二进制包,以及自动解决依赖项的) ca ...
- ubuntu下编译VLC
ubuntu下编译VLC 标签(空格分隔): ubuntu vlc 视频 编译 [TOC] 1.下载VLC源码包并解压 VLC的源码包在VLC的官网有,可以直接下载.也可以使用git来clone一个. ...
- 在Ubuntu下编译wpa_supplicant
最近在研究WiFi联盟的多屏互动协议Miracast,其中需要用到wpa_supplicant,编译中遇到了一些问题,经过一下午的折腾也都解决了,下面分享给各位. 一.编译需要用到三个库源码包,分别是 ...
- 在Ubuntu下编译Assimp库
在Ubuntu下编译Assimp库 怎样在Ubuntu下编译Assimp库?这是我曾经编译成功后做的笔记,供參考. 1.去以下的站点去下载Assimp库: http://assimp.sourcefo ...
- ubuntu下编译java程序
ubuntu下编译java程序 首先需要安装jdk,并配置好相应环境变量 下面以简单的HelloWorld为例 文件名为HelloWorld.java java代码: public class Hel ...
- Ubuntu下编译Bilibili/ijkplayer
在做Android客户端视频播放器的过程中熟悉ijkplayer.他在视频支持协议上要比Android原生的mediaplayer做的要好.因此,自己也基于Bilibili/ijkplayer的重新编 ...
- Ubuntu下编译SHTOOLS
SHTOOLS是使用Fortran语言写的一个专门用于处理球谐函数的一个开源库,更多的介绍请猛戳这里,关于这个库的安装和使用,都在官网上有详细的说明,虽然很详细,但是编译的时候还是比较费劲,下面将我在 ...
随机推荐
- 三、HTTP响应
HTTP消息是服务器和客户端之间交换数据的方式 有两种类型的消息: 请求--由客户端发送用来触发一个服务器上的动作 相应--来自服务器的应答 一.HTTP响应的构成 1.状态行 HTTP响应的起始行被 ...
- 【Dart学习】--Dart之数组(List)的相关方法总结
一,初始化List 非固定长度list var testList = List(); print(testList.length);//输出0 固定长度List var testList2 = Lis ...
- LDFLAGS 与 LDLIBS
今天下载了开源程序“贪吃蛇” (github地址 : https://github.com/taylorconor/tinytetris.git) 在编译时,出现如下错 linux:~/code/ot ...
- windows10 环境下的RabbitMQ安装步骤（图文）
第一步:下载并安装erlang 原因:RabbitMQ服务端代码是使用并发式语言Erlang编写的,安装Rabbit MQ的前提是安装Erlang. 下载地址:http://www.erlang.or ...
- Python - Django - Cookie 简单用法
home.html: <!DOCTYPE html> <html lang="en"> <head> <meta charset=&quo ...
- Redis常用数据类型及各种数据类型应用和实现方式
Redis常用数据类型: StringHashListSetSorted set 在具体描述这几种数据类型之前,我们先通过一张图了解下Redis内部内存管理中是如何描述这些不同数据类型的: 首先Red ...
- (NSNumber **)value和(NSNumber * __autoreleasing *)value
今天在看别人开源项目的时候看到这样的代码: 正文从这里开始~~~ 定义如下: /** 评论详情页基础设置 @param BaseSettingBlock 基础设置 */ - (void)setUpCo ...
- idea能用下划线替换红色报错吗？我色弱，用idea简直太痛苦了
看看下图的idea,如果某个类的包路径没有引进来,使用颜色来提示,这对于色弱的程序员简直是一种折磨,有没有可以改成eclipse的那种报错提示方式? 个人感觉idea真的没有eclipse友好,也许是 ...
- openpyxl,xlrd,win32com,wxpython,logging
目录 一. openpyxl常用操作总结 二. xlrd常用操作总结 三. win32com常用操作总结 四. 自定义异常 五. 判断中文 六. Excel数字字母转换 七. 使用wxpython进行 ...
- 用MATLAB的Classficiation Learner工具箱对12个数据集进行各种分类与验证
准备材料 以所有的特征集作为variable进行像Bayes吖.SVM吖.决策树吖......分类.同时对数据进行预处理,选出相关度高的特征子集作为新的一组data进行分类(预处理的代码不必放出来). ...