hadoop集群配置实例

1）ssh配置

http://allthingshadoop.com/2010/04/20/hadoop-cluster-setup-ssh-key-authentication/

2) 修改打开文件数限制  - 主从配置一致

/etc/security/limits.conf中增加如下配置提高打开文件限制数

hadoop soft nofile 10240

hadoop hard nofile 65536



3）配置core-site.xml  - 主从配置一致(配置文件在hadoop/conf目录，下同)

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

        <name>fs.default.name</name>

        <value>hdfs://server:6060/</value>

        <description>who is the namenode</description>

    </property>

</configuration>

4）配置hdfs-site.xml  - 主从配置一致

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

        <name>dfs.replication</name>

        <value>1</value>

        <final>true</final>

        <description>default 3 in production environment</description>

    </property>

    <property>

        <name>dfs.data.dir</name>

        <value>/home/hadoop/hdfs/data</value>

        <description>where data file is saved</description>

    </property>

    <property>

        <name>dfs.name.dir</name>

        <value>/home/hadoop/hdfs/name</value>

        <description>where data meta file is saved (on namenode) </description>

    </property>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>/home/hadoop/hdfs/tmp</value>

        <description>A base for other temporary directories.</description>

    </property>

</configuration>



目录权限

A）data目录权限

chmod 755 /home/hadoop/hdfs/data

B）namenode的数据目录不需要创建，hadoop namenode -format将自动创建

rm -rf /home/hadoop/hdfs/name



C）如果发现datanode也启动不了了

rm -rf /home/hadoop/hdfs/data



D）namenode几点显示只有一个datanode - 本机

从节点日志显示中显示

2013-11-04 15:37:29,138 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: server/192.168.16.133:6060. Already tried 0 time(s).

从节点无法连接到namenode，

修改方式：

在namenode上，hostname对应地址为127.0.0.1，需要修改/etc/hosts中的127.0.0.1（注释掉）

E) 如果主从仍旧无法连接，关闭防火墙



5）修改masters - 主从配置一致

server



6)修改slaves - 主动配置一致

server

slave1

slave2

7）mapred-site.xml配置  - 主动配置一致

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

        <name>mapred.job.tracker</name>

        <value>server:8021</value>

        <final>true</final>

    </property>

    <property>

        <name>mapred.local.dir</name>

        <value>/home/hadoop/mapred/local</value>

        <final>true</final>

    </property>

    <property>

        <name>mapred.system.dir</name>

        <value>/home/hadoop/mapred/system</value>

        <final>true</final>

    </property>

    <property>

        <name>mapred.tasktracker.map.tasks.maximum</name>

        <value>7</value>

        <final>true</final>

    </property>

    <property>

        <name>mapred.tasktracker.reduce.tasks.maximum</name>

        <value>7</value>

        <final>true</final>

    </property>

    <property>

        <name>mapred.child.java.opts</name>

        <value>-Xmx400m</value>

    </property>

</configuration>

8）hadoop-evn.sh中配置

export JAVA_HOME=/path/to/jdk

9）初始化、启动

$ bin/hadoop namenode -format
$ bin/hadoop start-all.sh

$ hadoop dfsadmin -report

Warning: $HADOOP_HOME is deprecated.



Configured Capacity: 288730824704 (268.9 GB)

Present Capacity: 188958699520 (175.98 GB)

DFS Remaining: 188958601216 (175.98 GB)

DFS Used: 98304 (96 KB)

DFS Used%: 0%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0



-------------------------------------------------

Datanodes available: 3 (3 total, 0 dead)



Name: 192.168.16.133:50010

Decommission Status : Normal

Configured Capacity: 119873626112 (111.64 GB)

DFS Used: 40960 (40 KB)

Non DFS Used: 7385841664 (6.88 GB)

DFS Remaining: 112487743488(104.76 GB)

DFS Used%: 0%

DFS Remaining%: 93.84%

Last contact: Mon Nov 04 16:03:57 CST 2013





Name: 192.168.16.113:50010

Decommission Status : Normal

Configured Capacity: 97833930752 (91.11 GB)

DFS Used: 28672 (28 KB)

Non DFS Used: 86688358400 (80.73 GB)

DFS Remaining: 11145543680(10.38 GB)

DFS Used%: 0%

DFS Remaining%: 11.39%

Last contact: Mon Nov 04 16:03:55 CST 2013





Name: 192.168.16.156:50010

Decommission Status : Normal

Configured Capacity: 71023267840 (66.15 GB)

DFS Used: 28672 (28 KB)

Non DFS Used: 5697925120 (5.31 GB)

DFS Remaining: 65325314048(60.84 GB)

DFS Used%: 0%

DFS Remaining%: 91.98%

Last contact: Mon Nov 04 16:03:56 CST 2013

Hadoop版本1.0.3