hadoop分布式安装过程

一、安装准备及环境说明

1、下载hadoop-1.2.1，地址：http://apache.spinellicreations.com/hadoop/common/stable/hadoop-1.2.1-bin.tar.gz

2、JDK版本：jdk1.6.0_35 （64位，必须是1.6）

3、操作系统：CentOS 6.4 64位

4、三台机器，10.108.102.5（master，hbase1），10.108.102.151（slave,hbase2），10.108.103.89（slave,hbase3）

二、安装操作

1、拷贝以上文件到Linux的“/home”目录下。同时新建目录“/home/songtao”。

2、安装JDK，此步省略...

3、解压hdaoop到/jz目录下。tar -zxvf  hadoop-1.2.1-bin.tar.gz  -C /home/songtao

4、配置 hbase1 机器可以通过SSH直接访问 hbase2 和 hbase3 。

在hbase1上执行以下操作：

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

直接回车，完成后会在~/.ssh/生成两个文件：id_dsa 和id_dsa.pub。这两个是成对出现，类似钥匙和锁。再把id_dsa.pub 追加到授权key 里面(当前并没有authorized_keys文件)：

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

完成后可以实现无密码登录本机：

$ ssh localhost

把hbase1上的id_dsa.pub 文件追加到hbase2和hbase3的authorized_keys 内( 以10.108.102.151节点为例)：

a. 拷贝hbase1的id_dsa.pub文件：

$ scp id_dsa.pub hbase2:/root/

b. 登录hbase2，进入/root目录执行：

$ cat id_dsa.pub >> .ssh/authorized_keys

之后可以在hbase1上不输入密码直接访问hbase2 ，同样方式配置无密码登录hbase3。

同样方法配置hbase2 hbase3的对其他机器无密码登录。

5、修改/home/songtao/hadoop-1.2.1/conf/hadoop-env.sh文件的环境变量：

# The java implementation to use.  Required.
export JAVA_HOME=/home/songtao/jdk1..0_35
export HADOOP_PID_DIR=/home/songtao/hadoop-1.2./pids

6、修改/home/songtao/hadoop-1.2.1/conf/core-site.xml配置文件，内容如下：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/hadoop_tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://hbase1:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

</configuration>

7、修改/home/songtao/hadoop-1.2.1/conf/hdfs-site.xml配置文件，内容如下：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>${hadoop.tmp.dir}/dfs/name</value>
  <description>Determines where on the local filesystem the DFS name node
  should store the name table(fsimage).  If this is a comma-delimited list
  of directories then the name table is replicated in all of the
  directories, for redundancy. </description>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>/home/songtao/hadoop-1.2.1/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>

<property>
  <name>dfs.replication</name>
  <value>2</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>
</configuration>

8、修改/home/songtao/hadoop-1.2.1/conf/mapred-site.xml配置文件，内容如下：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>hbase1:9001</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
</configuration>

9、修改ect/hosts配置文件,内容如下：

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost
10.108.102.5 hbase1
10.108.102.151 hbase2
10.108.103.89 hbase3

10、修改/ect/profile配置文件，在末尾追加以下内容，并输入source/etc/profile使之生效：

export JAVA_HOME=/usr/java/jdk1..0_35
export JRE_HOME=/usr/java/jdk1..0_35/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/home/songtao/hadoop-1.2.
export PATH=$HADOOP_HOME/bin:$PATH

11、将/home/songtao/hadoop-1.2.1拷贝到hbase2和hbase3对应的目录下。将/ect/profile和/etc/hosts也拷贝到hbase2和hbase3机器上。注意profile需要做生效操作。

$ source /etc/profile

三、格式化分布式文件系统hadoop

在名称节点的hadoop-1.1.2 目录下运行以下命令，出现sucessfully formatted表明格式化成功了

$ bin/hadoop namenode&nbsp -format

四、启动/停止hadoop

1、通过shell脚本启动hadoop。

$ sh /home/songtao/hadoop-1.2./bin/start-all.sh

或者分别启动

$ sh /home/songtao/hadoop-1.2./bin/start-dfs.sh
$ sh /home/songtao/hadoop-1.2./bin/start-mapred.sh

2、停止hadoop

$ sh /jz/hadoop-1.2./bin/stop-all.sh

五、判断是否成功

在名称节点和数据节点上分别 运行 /home/songtao/jdk1.6.0_35/bin/jps命令（查看与java有关的进行信息），出现下图几个进程表示安装成功！