Hadoop源码编译环境搭建

准备工具：

maven 3.0.0版本或者更高版本（配置中心库）

protocbuff 2.5.0

http://www.zlib.net/

git bash（Windows环境可以用此工具执行编译命令）

下载源码：

http://hadoop.apache.org/releases.html

http://mirror.bit.edu.cn/apache/hadoop/common/

当前稳定版本为hadoop 2.9.2

编译源码：

在目录hadoop-maven-plugins下执行maven命令

mvn clean package -Pdist -DskipTests

mvn clean install -DskipTests

mvn clean install -Dmaven.test.skip=true

在目录hadoop-2.9.2-src下执行

mvn eclipse:eclipse -DskipTests

或者

mvn eclipse:eclipse -Dmaven.test.skip=true

然而编译到最后，挂红了、、 

定位如下：

原来是hadoop-aws工程报错。尝试注掉该工程（注释掉该工程，不影响正常阅读Hadoop源码）：

重新编译：

共80个工程项目（hadoop-aws）失败

将Eclipse工程导入到IDEA，需要设置或更改M2_REPO和JDK版本（Hadoop 2.9.2使用JDK 1.7）

----------------------------------------------------------------------------------
Building on Windows
----------------------------------------------------------------------------------
Requirements:
* Windows System
* JDK 1.8
* Maven 3.0 or later
* ProtocolBuffer 2.5.
* CMake 3.1 or newer
* Visual Studio  Professional or Higher
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
  tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')
Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads
If using Visual Studio, it must be Professional level or higher.
Do not use Visual Studio Express.  It does not support compiling for -bit,
which is problematic if running a -bit system.
The Windows SDK 8.1 is available to download at:
http://msdn.microsoft.com/en-us/windows/bg162891.aspx
Cygwin is not required.
----------------------------------------------------------------------------------
Building:
Keep the source code tree in a short path to avoid running into problems related
to Windows maximum path length limitation (for example, C:\hdc).
There is one support command file located in dev-support called win-paths-eg.cmd.
It should be copied somewhere convenient and modified to fit your needs.
win-paths-eg.cmd sets up the environment for use. You will need to modify this
file. It will put all of the required components in the command path,
configure the bit-ness of the build, and set several optional components.
Several tests require that the user must have the Create Symbolic Links
privilege.
All Maven goals are the same as described above with the exception that
native code is built by enabling the 'native-win' Maven profile. -Pnative-win
is enabled by default when building on Windows since the native components
are required (not optional) on Windows.
If native code bindings for zlib are required, then the zlib headers must be
deployed on the build machine. Set the ZLIB_HOME environment variable to the
directory containing the headers.
set ZLIB_HOME=C:\zlib-1.2.
At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested
with zlib 1.2., built using Visual Studio  out of contrib\vstudio\vc10 in
the zlib 1.2. source tree.
http://www.zlib.net/
----------------------------------------------------------------------------------
Building distributions:
 * Build distribution with native code    : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar][-Dmaven.javadoc.skip=true]
----------------------------------------------------------------------------------
Running compatibility checks with checkcompatibility.py
Invoke `./dev-support/bin/checkcompatibility.py` to run Java API Compliance Checker
to compare the public Java APIs of two git objects. This can be used by release
managers to compare the compatibility of a previous and current release.
As an example, this invocation will check the compatibility of interfaces annotated as Public or LimitedPrivate:
./dev-support/bin/checkcompatibility.py --annotation org.apache.hadoop.classification.InterfaceAudience.Public --annotation org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate --include "hadoop.*" branch-2.7. trunk
----------------------------------------------------------------------------------
Changing the Hadoop version declared returned by VersionInfo
If for compatibility reasons the version of Hadoop has to be declared as a .x release in the information returned by
org.apache.hadoop.util.VersionInfo, set the property declared.hadoop.version to the desired version.
For example: mvn package -Pdist -Ddeclared.hadoop.version=2.11
If unset, the project version declared in the POM file is used.

----------------------------------------------------------------------------------
Building distributions:
Create binary distribution without native code and without documentation:
  $ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
Create binary distribution with native code and with documentation:
  $ mvn package -Pdist,native,docs -DskipTests -Dtar
Create source distribution:
  $ mvn package -Psrc -DskipTests
Create source and binary distributions with native code and documentation:
  $ mvn package -Pdist,native,docs,src -DskipTests -Dtar
Create a local staging version of the website (in /tmp/hadoop-site)
  $ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site
----------------------------------------------------------------------------------
Installing Hadoop
Look for these HTML files after you build the document by the above commands.
  * Single Node Setup:
    hadoop-project-dist/hadoop-common/SingleCluster.html
  * Cluster Setup:
    hadoop-project-dist/hadoop-common/ClusterSetup.html

README