单节点部署Hadoop教程

搭建HDFS

增加主机名

我这里仅仅增加了master主机名

[root@10 /xinghl/hadoop/bin]$ cat /etc/hosts
127.0.0.1   localhost 10.0.67.101
::1         localhost 10.0.67.101
10.0.67.101 master    10.0.67.101

如果配置远程集群，则需要配置SSH

我这里就是单节点测试玩玩，所以这步骤先省略了。

解压缩hadoop到/usr目录下

创建几个需要的目录

mkdir /dfs
mkdir /dfs/name
mkdir /dfs/data
mkdir /tmp

修改配置文件，在$HADOOP_HOME/etc/hadoop下

修改hadoop-env.sh

export JAVA_HOME=/usr/java

修改slaves

我这里就是

localhost

修改core-site.xml

<configuration>
       <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:8020</value>
       </property>
       <property>
                <name>io.file.buffer.size</name>
                <value>131072</value>
        </property>
       <property>
               <name>hadoop.tmp.dir</name>
               <value>file:/usr/hadoop/tmp</value>
               <description>Abase for other temporary   directories.</description>
       </property>
        <property>
               <name>hadoop.proxyuser.u0.hosts</name>
               <value>*</value>
       </property>
       <property>
               <name>hadoop.proxyuser.u0.groups</name>
               <value>*</value>
       </property>
</configuration>

修改hdfs-site.xml

<configuration>
       <property>
                <name>dfs.namenode.secondary.http-address</name>
               <value>master:9001</value>
       </property>
     <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/hadoop/dfs/name</value>
       </property>
      <property>
              <name>dfs.datanode.data.dir</name>
              <value>file:/usr/hadoop/dfs/data</value>
       </property>
       <property>
               <name>dfs.replication</name>
               <value>3</value>
        </property>
        <property>
                 <name>dfs.webhdfs.enabled</name>
                  <value>true</value>
         </property>
</configuration>

启动hadoop

hadoop namenode -format #在$HADOOP_HOME/bin目录下
start-all.sh #在$HADOOP_HOME/sbin目录下

检查运行状态

1 使用jps命令查看

2 登录http://10.0.67.101:8088/cluster

配置SSH，支持无密码登录

[root@localhost sbin]# cd ~/.ssh/
[root@localhost .ssh]# ll
total 0
[root@localhost .ssh]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
75:ea:4f:ec:df:55:06:3c:af:91:12:50:69:a5:86:27 root@localhost.localdomain
The key's randomart image is:
+--[ RSA 2048]----+
|          ...o.  |
|           ooo   |
|          E.* +  |
|         . * . = |
|        S . . o +|
|         . . . +.|
|          . o . .|
|           +   ..|
|            o.. .|
+-----------------+
[root@localhost .ssh]#
[root@localhost .ssh]# ll
total 8
-rw-------. 1 root root 1675 Aug 16 10:55 id_rsa
-rw-r--r--. 1 root root  408 Aug 16 10:55 id_rsa.pub
[root@localhost .ssh]# cat id_rsa.pub >> ~/.ssh/authorized_keys
[root@localhost .ssh]# ll
total 12
-rw-r--r--. 1 root root  408 Aug 16 10:55 authorized_keys
-rw-------. 1 root root 1675 Aug 16 10:55 id_rsa
-rw-r--r--. 1 root root  408 Aug 16 10:55 id_rsa.pub
[root@localhost .ssh]# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 9e:7b:cb:c5:06:e9:81:e5:db:57:9c:f9:79:a2:c4:ce.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
[root@localhost ~]# exit
logout
Connection to localhost closed.
[root@localhost .ssh]# ssh localhost
Last login: Tue Aug 16 10:55:30 2016 from localhost
[root@localhost ~]# exit
logout
Connection to localhost closed.
[root@localhost .ssh]#