hadoop-2.0.0-cdh4.6.0、sqoop-1.4.3-cdh4.6.0、mahout-0.7-cdh4.6.0 安装笔记

1. /etc/profile中环境变量配置:

export HADOOP_HOME=/usr/local/hadoop/cdh4.6.0/hadoop-2.0.
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

export CLASSPATH=$CLASSPATH:$HADOOP_CONF_DIR

export SQOOP_HOME=/usr/local/hadoop/cdh4.6.0/sqoop-1.4.
export PATH=$PATH:$SQOOP_HOME/bin

export MAHOUT_HOME=/usr/local/hadoop/cdh4.6.0/mahout-0.7
export PATH=$PATH:$MAHOUT_HOME/bin

环境变量

1.1  /etc/hosts 中加上：192.168.6.91    ds

2. $HADOOP_HOME/etc/hadoop目录下，修改配置文件

　　1）core-site.xml：

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://ds:9000</value>
  </property>
  <property>
    <name>fs.trash.interval</name>
    <value></value>
  </property>
  <property>
    <name>fs.trash.checkpoint.interval</name>
    <value></value>
  </property>
</configuration>

core-site.xml

　　2）yarn-site.xml：

<configuration>
<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>ds:</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>ds:</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>ds:</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>ds:</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>ds:</value>
  </property>
  <property>
    <description>Classpath for typical applications.</description>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
    $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
    $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
    $YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,
    $YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*
    </value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce.shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/usr/local/hadoop/cdh4.6.0/hadoop-2.0.0/yarn/local</value>
  </property>
  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/usr/local/hadoop/cdh4.6.0/hadoop-2.0.0/yarn/logs</value>
  </property>
  <property>
    <description>Where to aggregate logs</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/usr/local/hadoop/cdh4.6.0/hadoop-2.0.0/yarn/logs</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <value>/usr/local/hadoop/cdh4.6.0/hadoop-2.0.0</value>
</property>
</configuration>

yarn-site.xml

　　3）mapred-site.xml：

<configuration>
 <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>ds:</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>ds:</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>ds:</value>
  </property>

</configuration>

mapred-site.xml

4）hadoop-env.sh

　　最后加上export JAVA_HOME=/usr/local/jdk1.6.0_13  或者  export JAVA_HOME=$JAVA_HOME

3. 创建目录：$HADOOP_HOME/tmp/dfs/data

4. 启动hadoop：

　　1）hadoop namenode -format

　　2) start-dfs.sh

　　jps看DataNode、SecondaryNameNode、NameNode 是否都起来了，没起来的话查对应的log

　　3）start-yarn.sh

　　jps看NodeManager、ResourceManager 是否都起来了

5. 修改$SQOOP_HOME/bin/configure-sqoop

　　注释掉所有关于zookeeper和hbase的部分

6. 命令：sqoop help  如果出现usage: ... 则安装成功

7. 命令：mahout  如果出现Valid program names are: ... 则Mahout安装成功 （Mahout不需要更改配置文件）