Ubuntu 下编译 Hadoop-2.9.2

系统环境

系统： ubuntu-18.10-desktop-amd64

maven： Apache Maven 3.6.0

jdk: jdk_1.8.0_201

ProtocolBuffer： portoc-2.5.0

OpenSSL： OpenSSL 1.1.1

cmake： cmake3.12.1

hadoop： hadoop-2.9.2

安装所需环境

# 安装各种所需依赖库

$ sudo apt-get install autoconf automake libtool zlib1g-dev pkg-config libssl-dev libsnappy-dev libbz2-dev

# 安装ssh-server

$ sudo apt-get install ssh-server

# 安装openssl

$ sudo apt-get install openssl

# 安装cmake

$ sudo apt-get install cmake 

# 安装编译c++库

$ sudo apt-get install g++

# 安装protobuf

# 下载protobuf-2.5.0

$ wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

# 解压缩

$ tar -zxvf protobuf-2.5.0.tar.gz

# 进入protobuf解压缩目录配置

$ ./configur --prefix=/usr/local/protoc-2.5.0

# 编译安装

$ make && make install

# 配置到环境变量中~/.bash_profile

export PATH="/usr/local/protoc-2.5.0/bin:$PATH"

编译过程中可能遇到的问题

注意 libssl-dev安装换成 libssl1.0-dev版本。 我之前就是默认安转，在编译OpenSSL功能的时候一直卡主编译不通过。具体错误形式如下

[exec] make[2]: *** [CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o] Error 1

[exec] make[1]: *** [CMakeFiles/hadoop_static.dir/all] Error 2

[exec] make: *** [all] Error 2


# 移除libssl-dev

$ sudo apt-get remote libssl-dev

# 安装libssl1.0-dev

$ sudo apt-get install libssl1.0-dev

开始编译 hadoop

$ mvn package -Pdist,native -DskipTests -Dtar

命令执行后需要等待他下载编译所需的所有依赖过程可能会很久

[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:

[INFO]

[INFO] Apache Hadoop Main ................................. SUCCESS [  0.758 s]

[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.426 s]

[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.470 s]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.169 s]

[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.134 s]

[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.145 s]

[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.224 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.923 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [  2.653 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  1.823 s]

[INFO] Apache Hadoop Common ............................... SUCCESS [ 40.786 s]

[INFO] Apache Hadoop NFS .................................. SUCCESS [  2.925 s]

[INFO] Apache Hadoop KMS .................................. SUCCESS [  8.053 s]

[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.050 s]

[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 46.997 s]

[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 42.883 s]

[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  5.907 s]

[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 11.476 s]

[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  5.320 s]

[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.055 s]

[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 11.390 s]

[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.053 s]

[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.045 s]

[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.620 s]

[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 25.431 s]

[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  2.577 s]

[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.067 s]

[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  6.540 s]

[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 25.577 s]

[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.779 s]

[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 11.509 s]

[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  2.671 s]

[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 12.742 s]

[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.760 s]

[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.285 s]

[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  1.976 s]

[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  1.730 s]

[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  2.516 s]

[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 23.469 s]

[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  1.727 s]

[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.037 s]

[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.611 s]

[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.039 s]

[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.041 s]

[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.027 s]

[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  4.166 s]

[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.126 s]

[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 11.880 s]

[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  8.171 s]

[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.818 s]

[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  4.549 s]

[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.999 s]

[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  3.505 s]

[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.104 s]

[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  2.656 s]

[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  2.052 s]

[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  9.580 s]

[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.545 s]

[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.964 s]

[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.355 s]

[INFO] Apache Hadoop Rumen ................................ SUCCESS [  2.651 s]

[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.451 s]

[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.281 s]

[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  0.910 s]

[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.269 s]

[INFO] Apache Hadoop Pipes ................................ SUCCESS [  4.864 s]

[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.166 s]

[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.425 s]

[INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.589 s]

[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  1.737 s]

[INFO] Apache Hadoop Client ............................... SUCCESS [  4.883 s]

[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.535 s]

[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.363 s]

[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  2.404 s]

[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.732 s]

[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 17.061 s]

[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.035 s]

[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 42.773 s]

[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  2.183 s]

[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.229 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  08:19 min

[INFO] Finished at: 2019-05-08T17:23:30+08:00

[INFO] ------------------------------------------------------------------------

leone@leone-ubuntu:~/Downloads/hadoop-2.9.2-src$

大功告成编译后的hadoop放在hadoop-2.9.2-src\hadoop-dist\target目录下。

Ubuntu下编译 Hadoop2.9的更多相关文章

  1. 在Ubuntu下编译hadoop2.5.x

    在Ubuntu下编译hadoop2.5.x 参考博客:http://www.aboutyun.com/thread-8130-1-1.html 1 下载hadoop源码: (1) http://www ...

  2. Ubuntu下用hadoop2.4搭建集群（伪分布式）

    要真正的学习hadoop,就必需要使用集群,可是对于普通开发人员来说,没有大规模的集群用来測试,所以仅仅能使用伪分布式了.以下介绍怎样搭建一个伪分布式集群. 为了节省时间和篇幅,前面一些步骤不再叙述. ...

  3. ubuntu下编译caffe

    Ubuntu下编译caffe 纯粹是个人编译的记录.不用CUDA(笔记本是amd卡,万恶的nvidia):不手动编译依赖包(apt-get是用来干啥的?用来直接装二进制包,以及自动解决依赖项的) ca ...

  4. ubuntu下编译VLC

    ubuntu下编译VLC 标签(空格分隔): ubuntu vlc 视频 编译 [TOC] 1.下载VLC源码包并解压 VLC的源码包在VLC的官网有,可以直接下载.也可以使用git来clone一个. ...

  5. 在Ubuntu下编译wpa_supplicant

    最近在研究WiFi联盟的多屏互动协议Miracast,其中需要用到wpa_supplicant,编译中遇到了一些问题,经过一下午的折腾也都解决了,下面分享给各位. 一.编译需要用到三个库源码包,分别是 ...

  6. 在Ubuntu下编译Assimp库

    在Ubuntu下编译Assimp库 怎样在Ubuntu下编译Assimp库?这是我曾经编译成功后做的笔记,供參考. 1.去以下的站点去下载Assimp库: http://assimp.sourcefo ...

  7. ubuntu下编译java程序

    ubuntu下编译java程序 首先需要安装jdk,并配置好相应环境变量 下面以简单的HelloWorld为例 文件名为HelloWorld.java java代码: public class Hel ...

  8. Ubuntu下编译Bilibili/ijkplayer

    在做Android客户端视频播放器的过程中熟悉ijkplayer.他在视频支持协议上要比Android原生的mediaplayer做的要好.因此,自己也基于Bilibili/ijkplayer的重新编 ...

  9. Ubuntu下编译SHTOOLS

    SHTOOLS是使用Fortran语言写的一个专门用于处理球谐函数的一个开源库,更多的介绍请猛戳这里,关于这个库的安装和使用,都在官网上有详细的说明,虽然很详细,但是编译的时候还是比较费劲,下面将我在 ...

随机推荐

  1. tensorflow 如何看shape

    https://blog.csdn.net/yinxingtianxia/article/details/78121941 输入: x= tf.truncated_normal([, , ], dty ...

  2. Flutter 中的路由

    Flutter 中的路由通俗的讲就是页面跳转.在 Flutter 中通过 Navigator 组件管理路由导航. 并提供了管理堆栈的方法.如:Navigator.push 和 Navigator.po ...

  3. iframe子页面无法返回上一页的问题

    本文讨论的场景是ipad终端. 如题,因业务需要,需要使用iframe嵌套子页面.让外层始终保持一个socket连接,避免socket每跳转一个页面都要重新关闭建立连接的问题.但是这样问题来了,上线后 ...

  4. Spring Cloud Hystrix Dashboard的使用 5.1.3

      Hystrix除了可以对不可用的服务进行断路隔离外,还能够对服务进行实时监控.Hystrix可以实时.累加地记录所有关于HystrixCommand的执行信息,包括每秒执行多少.请求成功多少.失败 ...

  5. 线性回归:boston房价

    from sklearn.linear_model import LinearRegression,Lasso,Ridge from sklearn.datasets import load_bost ...

  6. 【计算机视觉】基于局部二值相似性模式（LBSP）的运动目标检测算法

    基于局部二值相似性模式(LBSP)的运动目标检测算法 kezunhai@gmail.com http://blog.csdn.net/kezunhai 本文根据论文:Improving backgro ...

  7. 【GStreamer开发】GStreamer播放教程09——数字音频传输

    目标 本教程展示GStreamer是如何进行数字音频传输的. 介绍 在常见的模拟格式外,高端的音频系统通常都接受数字格式,压缩的非压缩的都能接受.因为音频信号是从电脑传到音箱,用一种更有弹性的形态会更 ...

  8. List<E>

    List<E>——列表 有序,存储和读取的顺序是一致的 由整数索引 允许重复 add(int index,E element)——将元素插入指定位置 get(int index)——获取指 ...

  9. flask的信号使用

    from blinker import Namespace from datetime import datetime from flask import request namespace = Na ...

  10. eNSP基于接口地址池的dhcp服务

    拓扑图如下 基于接口的dhcp是最简单的一种 我们对路由器的两个端口分别设置ip地址为192.168.1.254 192.168.2.254 然后分别进入接口进行下一步配置 dhcp select i ...