CDH版本hadoop2.6伪分布式安装

1、基础环境配置

主机名	IP地址	角色	Hadoop用户
centos05	192.168.48.105	NameNode、ResourceManager、SecondaryNameNode、 DataNode、NodeManager	hadoop

1.1、关闭防火墙和SELinux

1.1.1、关闭防火墙

　　  略

1.1.2、关闭SELinux

　　  略

　　  注：以上操作需要使用root用户

1.2、hosts配置

　　

 | [root@centos05 ~]#  vim/etc/hosts
 | ##hadoop host####
 | 192.168.48.105  centos05

　　

 | [root@centos05 ~]#  vim /etc/sysconfig//network

3 | HOSTNAME=centos05

　　注：以上操作需要使用root用户，通过ping 主机名可以返回对应的IP即可

1.3、创建主机账号及配置无密码访问

　　

新建用户，建议用adduser命令
sudo adduser hadoop
passwd hadoop

输入密码后一直按回车即可，最后输入y确定。
在创建hadoop用户的同时也创建了hadoop用户组，下面我们把hadoop用户加入到hadoop用户组
输入
sudo usermod -a -G hadoop hadoop 

前面一个hadoop是组名，后面一个hadoop是用户名。完成后输入一下命令查询结果。
cat  /etc/group

然后再把hadoop用户赋予root权限，让他可以使用sudo命令
切换到可以root的用户输入
sudo gedit /etc/sudoers
sudo vi /etc/sudoers
在图形界面可以用第一个命令，是ubuntu自带的一个文字编辑器，终端命令界面使用第二个命令。有关vi编辑器的使用自行百度。

修改文件如下：
# User privilege specification
root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL
保存退出，hadoop用户就拥有了root权限

生成私钥和公钥
ssh-keygen -t rsa
拷贝公钥到主机（需要输入密码）
ssh-copy-id hadoop@hadoop
注：以上操作需要在hadoop用户，通过hadoop用户ssh到本机主机不需要密码即可

1.4、Java环境配置

1.4.1、下载JDK

　　略

1.4.2、安装java

　　略

2、安装hadoop

2.1、下载安装CDH版本的hadoop

　　下载链接：http://archive-primary.cloudera.com/cdh5/cdh/5/

2.2、安装配置hadoop

　　hadoop的安装配置使用hadoop用户操作；

• 创建目录，用于存放hadoop数据；

[hadoop@centos05 ~]$ mkdir -p /home/hadoop/app/hadoop/hdfs/{name,data}

2.2.1、配置core-site.xml

[hadoop@centos05 ~]$vim  /opt/hadoop/hadoop-2.6./etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/opt/hadoop/tmp</value>
    </property>
</configuration>

2.2.2、配置hdfs-site.xml

[hadoop@centos05 hadoop]$ vim /opt/hadoop/hadoop-2.6./etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value></value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/hadoop/hdfs/data</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

2.2.3、配置mapred-site.xml

[hadoop@centos05 hadoop]$cd /opt/hadoop/hadoop-2.6./etc/hadoop

[hadoop@centos05 hadoop]$cp mapred-site.xml.template mapred-site.xml

[hadoop@centos05 hadoop]$vim /opt/hadoop/hadoop-2.6./etc/hadoop/mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

2.2.4、配置yarn-site.xml

[hadoop@centos05 hadoop]$  vim /opt/hadoop/hadoop-2.6./etc/hadoop/yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

2.2.5、配置slaves

[hadoop@centos05 hadoop]$ vim /opt/hadoop/hadoop-2.6./etc/hadoop/slaves

centos05

2.2.6、配置hadoop-env

　　修改hadoop-env.sh文件的JAVA_HOME环境变量，操作如下：　　

[hadoop@centos05 hadoop]$ vim /opt/hadoop/hadoop-2.6./etc/hadoop/hadoop-env.sh

export JAVA_HOME=/opt/java/jdk1..0_191

2.2.7、配置yarn-env

　　修改yarn-env.sh文件的JAVA_HOME环境变量，操作如下：

[hadoop@centos05 hadoop]$ vim /opt/hadoop/hadoop-2.6./etc/hadoop/hadoop-env.sh

export JAVA_HOME=/opt/java/jdk1..0_191

2.2.8、配置mapred-env

　　修改mapred-env.sh文件的JAVA_HOME环境变量，操作如下：

[hadoop@centos05 hadoop]$ vim /opt/hadoop/hadoop-2.6./etc/hadoop/hadoop-env.sh

export JAVA_HOME=/opt/java/jdk1..0_191

2.2.9、配置HADOOP_PREFIX

　　配置HADOOP主机用户环境变量：

[hadoop@centos05 ~]$ vim .bash_profile

####HADOOP_PREFIX####
export HADOOP_PREFIX=/opt/hadoop/hadoop-2.6.
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin

　　启用环境变量

[hadoop@centos05 ~]$ source .bash_profile 

　　注：通过echo $HADOOP_PREFIX命令返回hadoop的安装目录

3、启动hadoop伪分布式

3.1、启动hdfs和yarn

• 格式化hdfs

  [hadoop@centos05 ~]$  hdfs namenode -format

• 启动dfs

• 启动yarn

  [hadoop@centos05 ~]$  start-dfs.sh
  
  [hadoop@centos05 ~]$  start-yarn.sh

  　　

• 查看启动的进程

  [hadoop@centos05 ~]$ jps
   DataNode
   ResourceManager
   SecondaryNameNode
   Jps
   NodeManager
   NameNode

  注：关闭dfs命令为：stop-dfs.sh     stop-yarn.sh

3.3、启动集群

　　hdfs和yarn的启动可以使用一条命令执行：　　

启动：start-all.sh
关闭:  stop-all.sh

• 启动后的所有进程：　　

[hadoop@centos05 ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [centos05]
centos05: starting namenode, logging to /opt/hadoop/hadoop-2.6./logs/hadoop-hadoop-namenode-centos05.out
centos05: starting datanode, logging to /opt/hadoop/hadoop-2.6./logs/hadoop-hadoop-datanode-centos05.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to 
　　　　　 /opt/hadoop/hadoop-2.6./logs/hadoop-hadoop-secondarynamenode-centos05.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop-2.6./logs/yarn-hadoop-resourcemanager-centos05.out
centos05: starting nodemanager, logging to /opt/hadoop/hadoop-2.6./logs/yarn-hadoop-nodemanager-centos05.out
[hadoop@centos05 ~]$ 

　

• 启动后的所有进程：

[hadoop@centos05 ~]$ jps
 NodeManager
 Jps
 NameNode
 ResourceManager
 SecondaryNameNode
 DataNode

• YARN管理界面：http://192.168.48.105:8088

• HDFS管理界面：http://192.168.48.105:50070

4、hdfs的shell操作和Wordcount演示

4.1、简单的hdfs shell操作

• 创建目录

  [hadoop@centos05 ~]$ hadoop fs -mkdir /input_test
  $ hadoop fs -mkdir /output_test

• 查看目录

  [hadoop@centos05 ~]$ hadoop fs -ls /
  Found  items
  drwxr-xr-x   - hadoop supergroup           -- : /input_test
  drwxr-xr-x   - hadoop supergroup           -- : /output_test
  drwx------   - hadoop supergroup           -- : /tmp

• 上传文件

  [hadoop@centos05 /]$ hadoop fs -put  /opt/hadoop/hadoop-2.6./share/doc/index.html  /input_test

• 查看上传文件

• [hadoop@centos05 /]$ hadoop fs -ls    /input_test/index.html
  -rw-r--r--    hadoop supergroup       -- : /input_test/index.html

  　　

• 查看文本文件内容

  [hadoop@centos05 /]$ hadoop fs -cat    /input_test/index.html

4.2、Wordcount

　　将HDFS上/input_text/index.html 使用hadoop内置Wordcount的jar包统计文档的Wordcount

• 启动测试

  [hadoop@centos05 /]$ hadoop jar /opt/hadoop/hadoop-2.6./share/hadoop/mapreduce/
                       hadoop-mapreduce-examples-2.6.-cdh5.15.1.jar wordcount 
  　　                  /input_test/index.html /output_test/runcount
  // :: INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:
  // :: INFO input.FileInputFormat: Total input paths to process :
  // :: INFO mapreduce.JobSubmitter: number of splits:
  // :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1543369969234_0002
  // :: INFO impl.YarnClientImpl: Submitted application application_1543369969234_0002
  // :: INFO mapreduce.Job: The url to track the job: 
                                        http://centos05:/proxy/application_1543369969234_0002/
  // :: INFO mapreduce.Job: Running job: job_1543369969234_0002
  // :: INFO mapreduce.Job: Job job_1543369969234_0002 running in uber mode : false
  // :: INFO mapreduce.Job:  map % reduce %
  // :: INFO mapreduce.Job:  map % reduce %
  // :: INFO mapreduce.Job:  map % reduce %
  // :: INFO mapreduce.Job: Job job_1543369969234_0002 completed successfully
  // :: INFO mapreduce.Job: Counters:
          File System Counters
                  FILE: Number of bytes read=
                  FILE: Number of bytes written=
                  FILE: Number of read operations=
                  FILE: Number of large read operations=
                  FILE: Number of write operations=
                  HDFS: Number of bytes read=
                  HDFS: Number of bytes written=
                  HDFS: Number of read operations=
                  HDFS: Number of large read operations=
                  HDFS: Number of write operations=
          Job Counters
                  Launched map tasks=
                  Launched reduce tasks=
                  Data-local map tasks=
                  Total time spent by all maps in occupied slots (ms)=
                  Total time spent by all reduces in occupied slots (ms)=
                  Total time spent by all map tasks (ms)=
                  Total time spent by all reduce tasks (ms)=
                  Total vcore-milliseconds taken by all map tasks=
                  Total vcore-milliseconds taken by all reduce tasks=
                  Total megabyte-milliseconds taken by all map tasks=
                  Total megabyte-milliseconds taken by all reduce tasks=
          Map-Reduce Framework
                  Map input records=
                  Map output records=
                  Map output bytes=
                  Map output materialized bytes=
                  Input split bytes=
                  Combine input records=
                  Combine output records=
                  Reduce input groups=
                  Reduce shuffle bytes=
                  Reduce input records=
                  Reduce output records=
                  Spilled Records=
                  Shuffled Maps =
                  Failed Shuffles=
                  Merged Map outputs=
                  GC time elapsed (ms)=
                  CPU time spent (ms)=
                  Physical memory (bytes) snapshot=
                  Virtual memory (bytes) snapshot=
                  Total committed heap usage (bytes)=
          Shuffle Errors
                  BAD_ID=
                  CONNECTION=
                  IO_ERROR=
                  WRONG_LENGTH=
                  WRONG_MAP=
                  WRONG_REDUCE=
          File Input Format Counters
                  Bytes Read=
          File Output Format Counters
                  Bytes Written=
  [hadoop@centos05 /]$ 

• 查看结果

  [hadoop@centos05 /]$ hadoop fs -ls /output_test/runcount/
  
  Found  items
  -rw-r--r--    hadoop supergroup           -- : /output_test/runcount/_SUCCESS
  -rw-r--r--    hadoop supergroup       -- : /output_test/runcount/part-r-
  
  [hadoop@centos05 /]$ hadoop fs -cat  /output_test/runcount/part-r-
  --
  <!--
  <!DOCTYPE
  </a>
  </body>
  </div>
  </head>
  </html>
  </li>
  </style>
  </ul>
  <a
  <body
  <div
  ......略

5、遇到的问题

5.1、WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

解决：导致该问题的改版本是因为${HADOOP_PREFIX}/lib/native目录没有lib库，解决办法是到hadoop官网下载2.6的包，把lib/native目录下的数据拷贝过去。

5.2、openssl: false Cannot load libcrypto.so (libcrypto.so: 无法打开共享对象文件: 没有那个文件或目录)!

解决：/usr/lib64/目录下做一个libcrypto.so软连
接

cd /usr/lib64/
ln -s /usr/lib64/libcrypto.so.1.0.1e libcrypto.so

• 使用命令export HADOOP_ROOT_LOGGER=DEBUG,console可以在终端上看到更详细的日志信息方便排查问题；
• 以上两个问题可以使用命令检查是否为true：hadoop checknative

注：${HADOOP_PREFIX}表示hadoop的安装目录，或者说是${HADOOP_HOME}

6、参考资料

http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.5/hadoop-project-dist/hadoop-common/SingleCluster.html