hadoop2.x编译安装

Build instructions for Hadoop (Hadoop 编译安装,参考hadoop源码包中BUILDING.txt文档)

----------------------------------------------------------------------------------
Requirements（编译需求，需要满足以下条件才能编译成功）:

* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

安装以上依赖条件：

yum groupinstall “Development tools”
yum -y install cmake zlib openssl gcc zlib-devel openssl-devel make

tar -zxvf protobuf-2.5.0.tar.gz
make && make install
protoc --version

安装maven（apache-maven-3.0.5-bin.zip）
解压、配置环境变量-略
[root@chavin ~]# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /usr/local/maven305
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_67/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
配置软件库：
[root@chavin ~]# ll /home/hadoop/.m2/
total 16
drwxrwxr-x. 56 hadoop hadoop  4096 Mar  3 20:13 repository
-rw-r--r--.  1 hadoop hadoop 10224 Mar  3 20:13 settings.xml
其中repository通过tar包repository.tar.gz解压得到。

编译前安装findbugs
下载：findbugs-3.0.0.tar.gz
解压:
tar  -zxvf findbugs-3.0.0.tar.gz

设置环境变量：
export FINDBUGS_HOME = /usr/local/findbugs
export PATH=$PATH:$FINDBUGS_HOME/bin

执行命令：source /etc/profile或者 . /etc/profile
验证：findbugs -version

----------------------------------------------------------------------------------
编译选项：
Build options:

* Use -Pnative to compile/bundle native code
  * Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
  * Use -Psrc to create a project source TAR.GZ
  * Use -Dtar to create a TAR with the distribution (using -Pdist)

Snappy build options:

Snappy is a compression library that can be utilized by the native code.
   It is currently an optional component, meaning that Hadoop can be built with
   or without this dependency.

* Use -Drequire.snappy to fail the build if libsnappy.so is not found.
    If this option is not specified and the snappy library is missing,
    we silently build a version of libhadoop.so that cannot make use of snappy.
    This option is recommended if you plan on making use of snappy and want
    to get more repeatable builds.

* Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy
    header files and library files. You do not need this option if you have
    installed snappy using a package manager.
  * Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library
    files.  Similarly to snappy.prefix, you do not need this option if you have
    installed snappy using a package manager.
  * Use -Dbundle.snappy to copy the contents of the snappy.lib directory into
    the final tar file. This option requires that -Dsnappy.lib is also given,
    and it ignores the -Dsnappy.prefix option.

OpenSSL build options:

OpenSSL includes a crypto library that can be utilized by the native code.
   It is currently an optional component, meaning that Hadoop can be built with
   or without this dependency.

* Use -Drequire.openssl to fail the build if libcrypto.so is not found.
    If this option is not specified and the openssl library is missing,
    we silently build a version of libhadoop.so that cannot make use of
    openssl. This option is recommended if you plan on making use of openssl
    and want to get more repeatable builds.
  * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto
    header files and library files. You do not need this option if you have
    installed openssl using a package manager.
  * Use -Dopenssl.lib to specify a nonstandard location for the libcrypto library
    files. Similarly to openssl.prefix, you do not need this option if you have
    installed openssl using a package manager.
  * Use -Dbundle.openssl to copy the contents of the openssl.lib directory into
    the final tar file. This option requires that -Dopenssl.lib is also given,
    and it ignores the -Dopenssl.prefix option.

Tests options:

* Use -DskipTests to skip tests when running the following Maven goals:
    'package',  'install', 'deploy' or 'verify'
  * -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....
  * -Dtest.exclude=<TESTCLASSNAME>
  * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java
-------------------------------------------------------------------------------------
编译方法：
Building distributions:

Create binary distribution without native code and without documentation:

$ mvn package -Pdist -DskipTests -Dtar

Create binary distribution with native code and with documentation:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

Create source distribution:

$ mvn package -Psrc -DskipTests

Create source and binary distributions with native code and documentation:

$ mvn package -Pdist,native,docs,src -DskipTests -Dtar

Create a local staging version of the website (in /tmp/hadoop-site)

$ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

----------------------------------------------------------------------------------

Handling out of memory errors in builds（编译过程中可能遇到的问题）

----------------------------------------------------------------------------------

If the build process fails with an out of memory error, you should be able to fix
it by increasing the memory used by maven -which can be done via the environment
variable MAVEN_OPTS.

Here is an example setting to allocate between 256 and 512 MB of heap space to
Maven

export MAVEN_OPTS="-Xms256m -Xmx512m"

----------------------------------------------------------------------------------

编译命令：mvn package -Pdist,native -DskipTests -Dtar

mvn package -DskipTests -Dtar -Dsnappy.lib=/usr/lib64/ -Dbundle.snappy