hadoop2.5.1搭建（一）

1.1配置

1.1.1修改hosts

vi /etc/hosts 

192.168.220.64  cluster4 

192.168.220.63 cluster3

1.2安装jdk

rpm安装

rpm -ivh jdk-7u17-linux-x64.rpm 

环境变量

vi /etc/profile
#set java environment
JAVA_HOME=/usr/java/jdk1..0_17
CLASSPATH=.:$JAVA_HOME/lib.tools.jar
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH

环境变量生效

source /etc/profile 

链接

ln -s -f /usr/java/jdk1..0_17/jre/bin/java
ln -s -f /usr/java/jdk1..0_17/bin/javac

测试

java -version

java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) -Bit Server VM (build 23.7-b01, mixed mode)

安装jdk报错，但是后续貌似不影响，暂时跳过。

[root@cluster3 java]# rpm -ivh jdk-7u17-linux-x64.rpm
Preparing...                ########################################### [%]
   :jdk                    ########################################### [%]
Unpacking JAR files...
        rt.jar...
Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/rt.pack
        jsse.jar...
Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/jsse.pack
        charsets.jar...
Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/charsets.pack
        tools.jar...
Error: Could not open input file: /usr/java/jdk1..0_17/lib/tools.pack
        localedata.jar...
Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/ext/localedata.pack

1.3配置ssh公钥密钥自动登录

[root@cluster3 ~]# cd .ssh
[root@cluster3 .ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
::2a:0f:ce::a6:9d:d7:ed:0b:4b::6b:3d: root@cluster3
[root@cluster3 .ssh]# cat id_rsa.pub >> authorized_keys
[root@cluster3 .ssh]# scp authorized_keys root@192.168.220.64:/root/.ssh/
root@192.168.220.64's password:
authorized_keys                               %       .4KB/s   :
[root@cluster3 .ssh]# ssh root@cluster3
Last login: Thu Oct  ::  from localhost.localdomain
[root@cluster3 ~]# ssh root@cluster4
Last login: Tue Oct  ::  from 192.168.220.1

2安装hadoop

解压hadoop

tar -zxf hadoop-2.5..tar.gz  

2.1修改core-site.xml

vi core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://cluster3:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/hadoop-2.5./tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value></value>
    </property>
</configuration>  

2.2修改hdfs-site.xml

vi hdfs-site.xml
<configuration>
    <property>
        <name>dfs.nameservices</name>
        <value>hadoop-cluster3</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///usr/local/hadoop/hadoop-2.5.1/dfs/name</value>
    </property>
    <property>
        <name>dfs.namenode.data.dir</name>
        <value>file:///usr/local/hadoop/hadoop-2.5.1/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value></value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration> 

2.3修改mapred-site.xml

cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobtracker.http.address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>cluster3:</value>
    </property>
</configuration>  

2.4修改yarn-site.xml

vi yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>cluster3:</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>cluster3:</value>
    </property>
</configuration> 

2.5修改slaves

vi slaves
cluster4

3.1修改JAVA_HOME

分别在文件hadoop-env.sh和yarn-env.sh中添加JAVA_HOME配置

vi hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1..0_17

vi yarn-env.sh

export JAVA_HOME=/usr/java/jdk1..0_17

3.2hadoop环境变量

登录Master，配置Hadoop环境变量。

vi /etc/profile

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.5.
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

source /etc/profile

3.3格式化文件系统

hdfs namenode -format  

3.4启动、停止服务

[root@cluster3 sbin]# start-dfs.sh

[root@cluster3 sbin]# start-yarn.sh

[root@cluster3 sbin]# stop-dfs.sh

[root@cluster3 sbin]# stop-yarn.sh

4验证

[root@cluster3 hadoop]# jps
 Jps
 ResourceManager
 NameNode

[root@cluster4 hadoop]# jps
 NodeManager
 DataNode
 Jps

查看日志

[root@cluster3 logs]# tail -n200 /usr/local/hadoop/hadoop-2.5./logs/yarn-root-resourcemanager-cluster3.log 

4.2浏览器访问：

http://192.168.220.63:50070

http://192.168.220.63:8088