Hadoop学习笔记——搭建

一搭建环境列表

操作系统：centos6.5 64位

JDK环境：jdk1.7.0_71

hadoop版本：社区版本2.7.2，hadoop-2.7.2-src.tar.gz

主机名	ip	角色	用户
master1	192.168.204.202	Namenode;secondary namenode;resourcemanager	hadoop
slave1	192.168.204.203	Datanode; nodemanager	hadoop
slave2	192.168.204.204	Datanode; nodemanager	hadoop

二操作系统环境准备

1设置主机名:hostname

vi  /etc/sysconfig/network

2设置防火墙

chkconfig iptables off

service iptables off

3关闭Selinux

vi /etc/sysconfig/selinux

SELINUX=disabled

[root@cloud001 Desktop]# hostname

[root@cloud001 Desktop]# ifconfig

[root@cloud001 Desktop]# service iptables status

[root@cloud001 Desktop]# sestatus

4安装jdk

配置环境变量

[root@master1 hadoopsolf]vim /etc/profile

JAVA_HOME=/usr/java/jdk1.7.0_71（根据实际情况修改）

CLASSPATH=.:$JAVA_HOME/lib.tools.jar

PATH=$JAVA_HOME/bin:$PATH

export JAVA_HOME CLASSPATH PATH

[root@ master1 hadoopsolf]source /etc/profile

三hadoop2.X软件编译环境准备

1下载http://apache.claz.org/hadoop/common/最新版

2准备编译环境

tar -zxvf hadoop-2.7.2-src.tar.gz得到hadoop-2.7.2-src文件夹。

进入hadoop-2.7.2-src文件夹，查看BUILDING.txt

cd  hadoop-2.7.2-src

vim  BUILDING.txt

可以看到编译所需的库或者工具。

3jdk

安装jdk；然后打开/etc/profile配置jdk环境变量

export  JAVA_HOME=/usr/java/jdk1.7.0_71

export  CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/tools.jar

export  JRE_HOME=/usr/java/jdk1.7.0_71

export  PATH=$PATH:$JRE_HOME/bin

source  /etc/profile

运行javac -version 查看状态

4安装各种库

yum -y install svn ncurses-devel gcc*

yum -y install lzo-devel zlib-develautoconf automake libtool cmake openssl-devel

5安装protobuf-2.5.0.tar.gz

tar zxvf protobuf-2.5.0.tar.gz进入protobuf-2.5.0依次执行

cd  protobuf-2.5.0

./configure

make

make install

查版本[root@master1 protobuf-2.5.0]# protoc  -version

6安装maven

下载http://maven.apache.org/download.cgi

tar -zxvf apache-maven-3.3.9-bin.tar.gz  -C  /hadoopsolf

然后打开/etc/profile配置环境变量

export MAVEN_HOME=/hadoopsolf/apache-maven-3.3.9

export MAVEN_OPTS="-Xms256m-Xmx512m"

export PATH=$PATH:$MAVEN_HOME/bin

source /etc/profile

查版本：[root@master1 protobuf-2.5.0]# mvn -version

7安装ant

下载：http://ant.apache.org/bindownload.cgi

tar -zxvf  apache-ant-1.9.6-bin.tar.gz  -C /hadoopsolf

然后打开/etc/profile配置环境变量

export ANT_HOME=/hadoopsolf/apache-ant-1.9.6

export PATH=$ANT_HOME/bin:$PATH

source /etc/profile

验证：[root@master1 protobuf-2.5.0]# ant -version

8安装findbugs

下载：http://findbugs.sourceforge.net/downloads.html

vim /etc/profile 文件末尾添加：

export FINDBUGS_HOME=/hadoopsolf/findbugs-3.0.1

export PATH=$PATH:$FINDBUGS_HOME/bin

source /etc/profile

验证：[root@master1 protobuf-2.5.0]# findbugs -version

9 hadoop2.X软件编译

mvn clean package -Pdist,native -DskipTests -Dtar

或者

mvn package -Pdist,native -DskipTests -Dtar

务必保持网络畅通，需要经过漫长的等待！

四安装配置hadoop2.X

1配置ssh免密码登录

1.1各主机配置hosts文件与主机名

vi  /etc/hosts

192.168.204.202  master1

192.168.204.203   slave1

192.168.204.204   slave2

vi  /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=master1  // slave1, slave2

1.2各主机设置静态ip,互ping

1.3配置master1到2个slave（slave1,slave2）的免密码登录（按顺序操作）

(1)首先检查各机器是否已经安装[root@master1Desktop]# rpm -qa|grep  ssh

已经安装

openssh-askpass-5.3p1-94.el6.x86_64

libssh2-1.4.2-1.el6.x86_64

openssh-5.3p1-94.el6.x86_64

openssh-server-5.3p1-94.el6.x86_64

openssh-clients-5.3p1-94.el6.x86_64

如果没有安装则 yum install  ssh

(2)在master1主机:

hadoop用户执行:ssh-keygen -t rsa下一步继续到结束

[hadoop@master1 ~]$ cd  /home/hadoop/.ssh/

[hadoop@master1 .ssh]$ ls

id_rsa  id_rsa.pub

[hadoop@master1 .ssh]$ cat id_rsa.pub>> authorized_keys

root用户执行[master1Desktop]# chmod  600  /home/hadoop/.ssh/authorized_keys

hadoop用户执行验证

[hadoop@master1 ~]$ ssh  master1

Last login: Mon Feb 22 22:23:16 2016 from master1

[hadoop@master1 ~]$

slave机器hadoop用户执行：mkdir-p  /home/hadoop/.ssh

master1主机hadoop用户执行传送

[hadoop@master1~]$

scp /home/hadoop/.ssh/authorized_keys hadoop@slave1:/home/hadoop/.ssh/

[hadoop@master1~]$

scp /home/hadoop/.ssh/authorized_keys hadoop@slave2:/home/hadoop/.ssh/

slave机器root用户执行：chmod  600 /home/hadoop/.ssh/authorized_keys

master1机器hadoop用户执行验证

[hadoop@master1 ~]$ ssh  slave1

[hadoop@master1 ~]$ ssh  slave2

2安装hadoop2.X

hadoop用户在master1操作：

2.1把编译好的hadoop2.X解压至目录

（自定义，我这里是/home/hadoop）

[hadoop@master1 ~]$ tar  -zxvf  /hadoopsolf/hadoop-2.7.2.tar.gz  -C  /home/hadoop

配置hadoop2.X的环境变量，修改~/.bash_profile

vi  /home/hadoop/.bash_profile

export HADOOP_HOME=/home/hadoop/hadoop-2.7.2

export HADOOP_MAPRED_HOME=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

export HADOOP_PID_DIR=/var/hadoop/pids

--- 注意

(root用户创建/var/hadoop/pids并赋予hadoop权限

mkdir   -p /var/hadoop/pids

chown  -R hadoop: hadoop /var/hadoop/pids

)

export PATH=$PATH:$HADOOP_HOME/bin

export JAVA_HOME=/usr/java/jdk1.7.0_71

export  CLASSPATH=.:$JAVA_HOME/lib.tools.jar

export  PATH=$JAVA_HOME/bin:$PATH

2.2配置Hadoop中基础目录

cd  /home/hadoop/hadoop-2.7.2

$ mkdir -p dfs/name

$ mkdir -p dfs/data

$ mkdir -p tmp

$ cd  etc/hadoop

2.3配置Hadoop中配置文件

需要配置的文件如下core-site.xml, hdfs-site.xml, mapred-site.xml, hadoop-env.sh，所有的文件均位于hadoop2.7.2/etc/hadoop下面，具体需要的配置如下：

core-site.xml 配置如下:

<property>

<name>hadoop.tmp.dir</name>

<value>file:/home/hadoop/hadoop-2.7.2/tmp</value>

<description>Abase for other temporary directories.</description>

</property>   //接收Client连接的RPC端口，用于获取文件系统metadata信息。

　　　　　　<property>

　　　　　  <name>fs.defaultFS</name>

　　　　　　<value>hdfs://master1:9000</value>

　　　　　　</property>

<property>

<name>io.file.buffer.size</name>

<value>131702</value>

</property>

hdfs-site.xml配置如下：

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>master1:9001</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/hadoop/hadoop-2.7.2/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/home/hadoop/hadoop-2.7.2/dfs/data</value>

</property>

<property>   //2份

<name>dfs.replication</name>

<value>2</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

mapred-site.xml配置如下：（cp mapred-site.xml.template  mapred-site.xml）　　

<property>

<name>mapreduce.framework.name</name>  // mapreduce运行在yarn上。

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>master1:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>master1:19888</value>

</property>

yarn-site.xml配置如下:

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>master1:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>master1:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>master1:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>master1:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>master1:8088</value>

</property>

Vi slaves:

slave1

slave2

vi  hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_71

vi  yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_71

2.4拷贝

从master1远程复制文件夹到slaveX机

scp -r /home/hadoop/hadoop-2.7.2  slave1:/home/hadoop

scp -r /home/hadoop/hadoop-2.7.2  slave2:/home/hadoop

3启动hadoop2.X

master1机器操作

3.1初始化HDFS系统

bin/hdfs  namenode  -format

3.2开启NameNode和DataNode守护进程

sbin/start-dfs.sh（此命令启动了namenode、secondaryNamenode以及datanode）

[hadoop@master1 sbin]$ ./start-dfs.sh

Starting namenodes on [master1]

master1: Error: JAVA_HOME is not set and could not be found.

slave2: Error: JAVA_HOME is not set and could not be found.

slave1: Error: JAVA_HOME is not set and could not be found.

Starting secondary namenodes [master1]

master1: Error: JAVA_HOME is not set and could not be found.

解决

vi /home/hadoop/hadoop-2.7.2/libexec/hadoop-config.sh
添加 export JAVA_HOME=/usr/java/jdk1.7.0_71

此时在master1上面运行的进程有：namenode secondarynamenode

slaveX上面运行的进程有：datanode

./sbin/start-yarn.sh (此命令启动了ResourceManager和NodeManager)

此时在master1上面运行的进程有：namenode secondarynamenode resourcemanager

slaveX上面运行的进程有：datanode  NodeManager

查看各进程

[hadoop@master1 hadoop-2.7.2]$ jps

8176 Jps

4356 ResourceManager

6277 NameNode

6429 SecondaryNameNode

3.3基本状态查看

查看帮助:

[hadoop@master1 hadoop-2.7.2]$./hdfs  –help

[hadoop@master1 hadoop-2.7.2]$./hdfs dfs –help

[hadoop@master1 bin]$ hdfs dfsadmin  -help

查看集群状态：./bin/hdfs dfsadmin –report

查看文件块组成：./bin/hdfs fsck / -files -blocks

可以通过登录Web控制台，查看HDFS集群状态：     http://master1:50070 （hdfs-site.xml）

ResourceManager运行在主节点master上，查看yarn:    http://master1:8088 （yarn-site.xml）

NodeManager运行在从节点上，查看例如节点slave1：http://slave1:8042/

管理JobHistory Server(先要启动mr-jobhistory-daemon.shstart historyserver)，通过Web查看：http://master1:19888/jobhistory

查看hadoop：http://master1:9001