Uubntu14.04下 Hadoop集群环境搭建

1机器信息

master192.168.3.10 hadoop/ hadoop

slave1192.168.3.11 hadoop/ hadoop

slave2192.168.3.12 hadoop/ hadoop

（1）创建用户

#sudo adduser hadoop

（2）增加root权限

# vim /etc/sudoers

hadoop ALL=(ALL) ALL

（3）修改以上三台主机的/etc/hosts配置信息（内容一样）

192.168.3.10        hd-master

192.168.3.11        hd-slave1

192.168.3.12        hd-slave2

（4）分别修改/etc/hostname配置文件

2设置master到slave节点的SSH无密码登陆

Ubuntu已经默认安装了SSH

查看是否运行：ps -ef | grep sshd

（1）首先生成hd-master公钥

hadoop@hd-master:~$ ssh localhost

hadoop@hd-master:~/.ssh$ ssh-keygen -t rsa

hadoop@hd-master:~/.ssh$ cat id_rsa.pub >> authorized_keys

（2）将hd-master公钥保存到hd-slave1节点

hadoop@hd-master:~/.ssh$ scp id_rsa.pub hadoop@hd-slave1:/home/hadoop/

（3）在 hd-slave1节点 上将ssh公匙保存到相应位置

hadoop@hd-slave1:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys

（4）同样将hd-master公钥保存到hd-slave2节点，并在hd-slave2节点 上将ssh公匙保存到相应位置

hadoop@hd-master:~/.ssh$ scp id_rsa.pub hadoop@hd-slave2:/home/hadoop/

hadoop@hd-slave2:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys

（5）最后就可以在hd-master节点上就可以无密码SSH登录到hd-slave1、hd-slave2

hadoop@hd-master:~/.ssh$ ssh hd-slave1

hadoop@hd-master:~/.ssh$ ssh hd-slave2

3安装JDK

hadoop@hd-master:~$ java -version

java version "1.8.0_91"

hadoop@hd-slave1:~$ java -version

java version "1.8.0_91"

hadoop@hd-slave2:~$ java -version

java version "1.7.0_40"

4关闭防火墙

查看防护墙状态是否为“不活动”

hadoop@hd-master:~$ sudo ufw status

Status: inactive

5安装hadoop-2.7.3.tar.gz

（1）解压

hadoop@hd-master:~$ tar -zxvf hadoop-2.7.3.tar.gz

（2）在hd-master节点创建以下目录，并属于同一用户及用户组

hadoop@hd-master:~$ mkdir -p  ~/dfs/name && mkdir ~/dfs/data && mkdir ~/tmp

（3）在hd-master节点修改配置文件

A: hadoop-env.sh

export JAVA_HOME=${JAVA_HOME}

B: yarn-env.sh

export JAVA_HOME=${JAVA_HOME}

C: slaves保存所有slave节点

hd-slave1

hd-slave2

D: core-site.xml

<property>

<name>fs.defaultFS</name>

<value>hdfs://hd-master:8010</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>131072</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/home/hadoop/tmp</value>

<description>Abase for other temporary   directories.</description>

</property>

<property>

<name>hadoop.proxyuser.hadoop.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.groups</name>

<value>*</value>

</property>

E: hdfs-site.xml

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>hd-master:9001</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/hadoop/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/home/hadoop/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

F: mapred-site.xml

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>hd-master:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>hd-master:19888</value>

</property>

G: yarn-site.xml

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>hd-master:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>hd-master:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>hd-master:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>hd-master:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>hd-master:8088</value>

</property>

（4）将hd-master节点中的hadoop-2.7.3/复制到hd-slave1与hd-slave2节点。

hadoop@hd-master:~$ scp -r hadoop-2.7.3 hadoop@hd-slave1:/home/hadoop/

hadoop@hd-master:~$ scp -r hadoop-2.7.3 hadoop@hd-slave2:/home/hadoop/

（5）配置环境变量

hadoop@hd-master:~$ cat /etc/environment

PATH="/home/hadoop/hadoop-2.7.3/bin:/home/hadoop/hadoop-2.7.3/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"

（6）格式化namenode节点

hadoop@hd-master:~$ hdfs namenode -format

（7）启动hdfs

hadoop@hd-master:~$ start-dfs.sh

Starting namenodes on [hd-master]

hd-master: namenode running as process 7177. Stop it first.

hd-slave1: datanode running as process 2958. Stop it first.

hd-slave2: datanode running as process 10759. Stop it first.

Starting secondary namenodes [hd-master]

hd-master: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-hd-master.out

（8）启动yarn

hadoop@hd-master:~$ start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-resourcemanager-hd-master.out

hd-slave1: starting nodemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-hd-slave1.out

hd-slave2: starting nodemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-hd-slave2.out

hadoop@hd-master:~$

（9）查看进程

hadoop@hd-master:~$ jps

8324 ResourceManager

7177 NameNode

8667 Jps

8173 SecondaryNameNode

hadoop@hd-slave1:~$ jps

3335 NodeManager

3452 Jps

2958 DataNode

hadoop@hd-slave2:~$ jps

10759 DataNode

13192 Jps

12529 NodeManager

（10）查看集群信息

http://192.168.3.10:8088/cluster

1机器信息

master	192.168.3.10 hadoop/ hadoop
slave1	192.168.3.11 hadoop/ hadoop
slave2	192.168.3.12 hadoop/ hadoop