在Ubuntu14.10中部署Hadoop2.6.0单节点伪分布集群

1. 环境信息如下：

ubuntu：14.10

jdk：openjdk-1.7.0

hadoop：2.6.0

2. 下载hadoop2.6.0, http://apache.fayea.com/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

本文的$HADOOP_HOME为：/media/GoldenResources/localSoft/hadoop-2.6.0。

在 ~/.bashrc文件中添加环境变量，如下：

export HADOOP_HOME=/media/GoldenResources/localSoft/hadoop-2.6.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

然后使配置生效，命令如下：

source ~/.bashrc

3. 安装JDK，并设置JAVA_HOME环境变量。在/etc/profile文件最后添加如下内容

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386   //根据自己Java安装目录而定
export PATH=$JAVA_HOME/bin:$PATH

输入以下命令使配置生效

source /etc/profile

4. 配置SSH。首先生成秘钥，命令如下，然后一路回车确认，不需要任何输入。

ssh-keygen -t rsa

然后把公钥写入authorized_keys文件中，命令如下：

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

最后，输入下述命令，按提示输入 yes 即可，验证无密码访问是否成功。

ssh localhost

5. 修改Hadoop配置文件，进入${HADOOP_HOME}/etc/hadoop/目录。
1）修改core-site.xml，添加如下内容。

<property>
	<name>fs.default.name</name>
	<value>hdfs://localhost:9000</value>
</property>

2）修改hdfs-site.xml，添加以下内容。

<property>
    <name>dfs.repliacation</name>
     <value>1</value>
</property>

3）将mapred-site.xml.template重命名为mapred-site.xml，并添加如下内容。

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

4）修改yarn-site.xml，添加以下内容。

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

6. 格式化HDFS，命令如下：

bin/hdfs namenode -format 注释：bin/hadoop namenode -format命令已过时

默认情况下，格式化成功会在/tmp/hadoop-${user.name}目录下创建dfs文件夹。

7.启动HDFS，命令如下：

sbin/start-dfs.sh

出现以下错误：

$sbin/start-dfs.sh
15/03/16 21:36:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [OpenJDK Server VM warning: You have loaded library /media/GoldenResources/localSoft/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
localhost]
sed: -e 表达式 #1, 字符 8: “s”的未知选项↵
Server: ssh: Could not resolve hostname server: Name or service not known
-c: Unknown cipher type 'cd'
disabled: ssh: Could not resolve hostname disabled: Name or service not known
or: ssh: Could not resolve hostname or: Name or service not known
which: ssh: Could not resolve hostname which: Name or service not known
now.: ssh: Could not resolve hostname now.: Name or service not known
will: ssh: Could not resolve hostname will: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
'execstack: ssh: Could not resolve hostname 'execstack: Name or service not known
recommended: ssh: Could not resolve hostname recommended: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
OpenJDK: ssh: Could not resolve hostname openjdk: Name or service not known
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Name or service not known
'-z: ssh: Could not resolve hostname '-z: Name or service not known
It's: ssh: Could not resolve hostname it's: Name or service not known
<libfile>',: ssh: Could not resolve hostname <libfile>',: Name or service not known
it: ssh: Could not resolve hostname it: Name or service not known
link: ssh: Could not resolve hostname link: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known

参考 http://blog.csdn.net/xin_jmail/article/details/40556267中的解决方法，设置 HADOOP_COMMON_LIB_NATIVE_DIR和HADOOP_OPTS环境变量，在 ~/.bashrc文件中添加如下内容并编译。

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

source ~/.bashrc

重新启动HDFS，输出如下，表示启动成功。

$start-dfs.sh
15/03/16 22:21:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /media/GoldenResources/localSoft/hadoop-2.6.0/logs/hadoop-qunengrong-namenode-qunengrong-Studio-1450.out
localhost: starting datanode, logging to /media/GoldenResources/localSoft/hadoop-2.6.0/logs/hadoop-qunengrong-datanode-qunengrong-Studio-1450.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /media/GoldenResources/localSoft/hadoop-2.6.0/logs/hadoop-qunengrong-secondarynamenode-qunengrong-Studio-1450.out
15/03/16 22:21:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

可通过http://localhost:50070/ 来查看NameNode的运行状态

8. 启动Yarn，命令如下：

sbin/start-yarn.sh

可通过http://localhost:8088/cluster 来查看Application的执行状态