hadoop环境搭建笔记

一、配置Linux

（1）cat /etc/networks

（2）cat /etc/sysconfig/network

（3）vi /etc/udev/rules.d/70-persistent-net.rules

eth1 改为eth0

（4）vi /etc/sysconfig/network-scripts/ifcfg-eth0

HWADDR改为eth1的地址

HWADDR=00:0c:29:79:e6:ec

DNS1=网关地址

（5）vi /etc/sysconfig/selinux

#SELINUX=disabled

（6）加sudo权限

vi /etc/sudoers

beifeng ALL=(root)NOPASSWD:ALL

（7）关闭防火墙

centos 7：
systemctl stop firewalld.service #停止
systemctl disable firewalld.service #禁用
之前的版本：
service iptables stop #

停止
chkconfig iptables off #禁用

chkconfig iptables off

service iptables status

二、安装jdk、maven、hadoop

chown beifeng:beifeng /opt/*

无密钥登陆ssh
ssh-keygen -t rsa
ssh-copy-id hostname02
ssh-copy-id hostname03
ssh-copy-id hostname04

三、配置伪分布式

1. 修改hadoop-evn.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

2. 修改yarn-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

3. 修改mapred-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

4. 修改core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
     <property>
         <name>hadoop.tmp.dir</name>
         <value>/opt/modules/hadoop-2.5.0/data/tmp</value>
     </property>
     <property>
          <name>hadoop.http.staticuser.user</name>
          <value>beifeng</value>
     </property>
</configuration>

5. 修改hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>beifeng-hadoop-02:50090</value>
        </property>

        <property>
                <name>dfs.permissions.enabled</name>
                <value>false</value>
        </property>
</configuration>

6. 修改slaves

beifeng-hadoop-02

7. 修改yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>

        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>beifeng-hadoop-02</value>
        </property>

        <!-- 是否启用日志聚集功能 -->
        <property>
                <name>yarn.log-aggregation-enable</name>
                <value>true</value>
        </property>

        <!-- 日志保留时间(单位为秒) -->
        <property>
                <name>yarn.log-aggregation.retain-seconds</name>
                <value>106800</value>
        </property>
</configuration>

8. 修改mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

9. 启动服务

bin/hdfs namenode -format

sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start datanode
sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh start nodemanager
sbin/mr-jobhistory-daemon.sh start historyserver
sbin/hadoop-daemon.sh start secondarynamenode

sbin/hadoop-daemon.sh stop namenode
sbin/hadoop-daemon.sh stop datanode
sbin/yarn-daemon.sh stop resourcemanager
sbin/yarn-daemon.sh stop nodemanager
sbin/mr-jobhistory-daemon.sh stop historyserver
sbin/hadoop-daemon.sh stop secondarynamenode

10. 服务可视化

hdfs可视化界面： http://beifeng-hadoop-01:50070/dfshealth.html#tab-overview
yarn可视化界面： http://beifeng-hadoop-01:8088/cluster
历史服务器可视化界面：http://beifeng-hadoop-01:19888/
secondarynamenode可视化界面 http://beifeng-hadoop-01:50090/status.html