64位Linux编译hadoop-2.5.1

Apache Hadoop生态系统安装包下载地址：http://archive.apache.org/dist/

软件安装目录：~/app

jdk: jdk-7u45-linux-x64.rpm
hadoop: hadoop-2.5.-src.tar.gz
maven: apache-maven-3.0.-bin.zip
protobuf: protobuf-2.5..tar.gz

1、下载hadoop

wget http://archive.apache.org/dist/hadoop/core/stable/hadoop-2.5.1-src.tar.gz
tar -zxvf hadoop-2.5.-src.tar.gz 

在解压后的hadoop根目录下有个BUILDING.txt文件，可以看到编译hadoop的环境要求

Requirements:
* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

2、安装jdk

sudo yum install jdk-7u45-linux-x64.rpm

查看jdk安装位置：

which java
/usr/java/jdk1..0_45/bin/java

添加jdk到环境变量(~/.bash_profile)：

export JAVA_HOME=/usr/java/jdk1..0_45
export PATH=.:$JAVA_HOME/bin:$PATH

验证：

java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) -Bit Server VM (build 24.45-b08, mixed mode)

3、安装maven

wget http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.zip
unzip apache-maven-3.0.-bin.zip

添加maven到环境变量(~/.bash_profile):

export MAVEN_HOME=/home/hadoop/app/apache-maven-3.0.
export PATH=.:$MAVEN_HOME/bin:$PATH

验证：

mvn -version
Apache Maven 3.0. (r01de14724cdef164cd33c7c8c2fe155faf9602da; -- ::-)
Maven home: /home/hadoop/app/apache-maven-3.0.
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1..0_45/jre
Default locale: en_US, platform encoding: UTF-
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

4、安装protobuf

protobuf的官方地址貌似上不了，自行下载protobuf安装包；为了编译安装protobuf，需要先gcc/gcc-c++/make

sudo yum install gcc
sudo yum install gcc-c++
sudo yum install make

tar -zvxf protobuf-2.5..tar.gz
cd protobuf-2.5.
./configure --prefix=/usr/local/protoc/
sudo make
sudo make install

添加protobuf到环境变量(~/.bash_profile):

export PATH=.:/usr/local/protoc/bin:$PATH

验证：

protoc --version
libprotoc 2.5.

5、安装其他依赖

sudo yum install cmake
sudo yum install openssl-devel
sudo yum install ncurses-devel

6、编译hadoop源代码

cd ~/app/hadoop-2.5.-src
mvn package -DskipTests -Pdist,native

[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [.980s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [.575s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [.324s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [.318s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [.550s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [.548s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [.410s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [.503s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [.915s]
[INFO] Apache Hadoop Common .............................. SUCCESS [:.913s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [.324s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [.064s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [:.023s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [.389s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [.235s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [.493s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [.041s]
[INFO] hadoop-yarn ....................................... SUCCESS [.031s]
[INFO] hadoop-yarn-api ................................... SUCCESS [:.828s]
[INFO] hadoop-yarn-common ................................ SUCCESS [.542s]
[INFO] hadoop-yarn-server ................................ SUCCESS [.047s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [.953s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [.537s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [.270s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [.840s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [.877s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [.421s]
[INFO] hadoop-yarn-client ................................ SUCCESS [.406s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [.025s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [.208s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [.885s]
[INFO] hadoop-yarn-site .................................. SUCCESS [.058s]
[INFO] hadoop-yarn-project ............................... SUCCESS [.870s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [.065s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [.292s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [.197s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [.229s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [.322s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [.640s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [.154s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [.939s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [.088s]
[INFO] hadoop-mapreduce .................................. SUCCESS [.979s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [.615s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [.668s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [.014s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [.567s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [.398s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [.151s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [.251s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [.901s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [.722s]
[INFO] Apache Hadoop Client .............................. SUCCESS [.021s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [.095s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [.776s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [.768s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [.035s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [.571s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: :.071s
[INFO] Finished at: Sat Nov  :: PST
[INFO] Final Memory: 91M/324M
[INFO] ------------------------------------------------------------------------

编译后的代码在hadoop-2.5.1-src/hadoop-dist/target/hadoop-2.5.1下，以后要搭建hadoop环境直接使用hadoop-2.5.1文件夹部署即可。