【hadoop】hadoop3.2.0的安装并测试

前言：前段时间将hadoop01的虚拟机弄的崩溃掉了，也没有备份，重新从hadoop02虚拟上克隆过来的，结果hadoop-eclipse插件一样的编译，居然用不起了，找了3天的原因，最后还是没有解决，只能用hadoop shell 命令去测试了，反正影响不大，只不过用着不方便而已。

心累中...........

正文：

解压安装Hadoop

[hadoop@hadoop01 ~]$ cp /home/hadoop/Resources/hadoop-3.2.0.tar.gz ~/
[hadoop@hadoop01 ~]$ tar -zxvf ~/hadoop-3.2.0.tar.gz
[hadoop@hadoop01 ~]$ cd hadoop-3.2.0
[hadoop@hadoop01 hadoop-3.2.0]$ ls -l
total 184
drwxr-xr-x. 2 hadoop hadoop    203 Jan  8  2019 bin
drwxr-xr-x. 3 hadoop hadoop     20 Jan  8  2019 etc
drwxr-xr-x. 2 hadoop hadoop    106 Jan  8  2019 include
drwxr-xr-x. 3 hadoop hadoop     20 Jan  8  2019 lib
drwxr-xr-x. 4 hadoop hadoop   4096 Jan  8  2019 libexec
-rw-rw-r--. 1 hadoop hadoop 150569 Oct 19  2018 LICENSE.txt
-rw-rw-r--. 1 hadoop hadoop  22125 Oct 19  2018 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop   1361 Oct 19  2018 README.txt
drwxr-xr-x. 3 hadoop hadoop   4096 Jan  8  2019 sbin
drwxr-xr-x. 4 hadoop hadoop     31 Jan  8  2019 share

配置Hadoop环境变量

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/hadoop-env.sh
编辑文件并保存：
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
# export JAVA_HOME=
export JAVA_HOME=/usr/java/jdk1.8.0_11/

配置YARN环境变量

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/yarn-env.sh
编辑文件并保存：
export JAVA_HOME=/usr/java/jdk1.8.0_11/

配置核心组件文件（core-site.xml）

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/core-site.xml
编辑文件并保存：
<configuration>
<!--hdfs 的默认地址、端口 访问地址-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop01:9802</value>
</property>
<!--hdfs临时路径-->
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoopdata</value>
</property>
</configuration>

配置文件系统（hdfs-site.xml）

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/hdfs-site.xml
编辑文件并保存：
<configuration>
<!--hdfs web的地址 -->
<property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop01:50070</value>
</property>
<!-- 副本数-->
<property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
<!-- 是否启用hdfs权限检查 false 关闭 -->
       <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
<!-- 块大小,默认字节, 可使用 k m g t p e-->
       <property>
        <name>dfs.blocksize</name>
    <!--128m-->
        <value>134217728</value>
    </property>
    <!--hadoop的name和data目录路径-->
       <property>
         <name>dfs.namenode.name.dir</name>
         <value>file:/home/hadoop/hdfs/name</value>
   </property>
   <property>
         <name>dfs.datanode.data.dir</name>
         <value>file:/home/hadoop/hdfs/data</value>
   </property>
</configuration>

配置yarn-site.xml文件

[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/yarn-site.xml
编辑文件并保存：
<configuration>
<!-- Site specific YARN configuration properties -->
<!--集群master，-->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop01</value>
</property>

<!-- NodeManager上运行的附属服务-->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<!--容器可能会覆盖的环境变量，而不是使用NodeManager的默认值-->
<property>
        <name>yarn.nodemanager.env-whitelist</name>
    <value> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>
</property>
<!-- 关闭内存检测，虚拟机需要，不配会报错-->
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
</configuration>

配置MapReduce计算框架文件
[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/mapred-site.xml
编辑文件并保存：

<configuration>
<!--local表示本地运行，classic表示经典mapreduce框架，yarn表示新的框架-->
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<!--如果map和reduce任务访问本地库（压缩等），则必须保留原始值
当此值为空时，设置执行环境的命令将取决于操作系统：
Linux：LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
windows：PATH =%PATH%;%HADOOP_COMMON_HOME%\\bin.
-->
<property>
        <name>mapreduce.admin.user.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<!--
可以设置AM【AppMaster】端的环境变量
如果上面缺少配置，可能会造成mapreduce失败
-->
<property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>

【选】配置slaves文件（hadoop2.x修改slaves）
[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/slaves
编辑文件并保存：

hadoop02
hadoop03

【选】配置workers文件（hadoop3.x修改workers）
[hadoop@hadoop01 hadoop-3.2.0]$ gedit /home/hadoop/hadoop-3.2.0/etc/hadoop/workers
编辑文件并保存：

hadoop01
hadoop02
hadoop03

复制hadoop01上的Hadoop到hadoop02和hadoop03节点上
scp -r /home/hadoop/hadoop-3.2.0 hadoop@hadoop02:~/
scp -r /home/hadoop/hadoop-3.2.0 hadoop@hadoop03:~/

配置操作系统环境变量（需要在所有节点上进行，且使用一般用户权限）
gedit ~/.bash_profile
source ~/.bash_profile
编辑文件并保存：

#以下是新添加入代码
export JAVA_HOME=/usr/java/jdk1.8.0_11/
export PATH=$JAVA_HOME/bin:$PATH
#hadoop
export HADOOP_HOME=/home/hadoop/hadoop-3.2.0
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

创建Hadoop数据目录（所有节点操作）
mkdir /home/hadoop/hadoopdata

格式化文件系统（主端进行）
hdfs namenode -format

启动和关闭Hadoop
cd ~/hadoop-3.2.0
sbin/start-all.sh
stop-all.sh

启动成功结果：

[hadoop@hadoop01 hadoop-3.2.0]$ jps
20848 DataNode
21808 Jps
21076 SecondaryNameNode
21322 ResourceManager
20668 NameNode
21468 NodeManager
[hadoop@hadoop01 hadoop-3.2.0]$

【最后测试】在Hadoop集群中运行程序
将计算圆周率pi的Java程序包投入运行

[hadoop@hadoop01 hadoop-3.2.0]$ cd ~/hadoop-3.2.0/share/hadoop/mapreduce
[hadoop@hadoop01 mapreduce]$ ls
hadoop-mapreduce-client-app-3.2.0.jar     hadoop-mapreduce-client-hs-plugins-3.2.0.jar       hadoop-mapreduce-client-shuffle-3.2.0.jar   lib
hadoop-mapreduce-client-common-3.2.0.jar  hadoop-mapreduce-client-jobclient-3.2.0.jar        hadoop-mapreduce-client-uploader-3.2.0.jar  lib-examples
hadoop-mapreduce-client-core-3.2.0.jar    hadoop-mapreduce-client-jobclient-3.2.0-tests.jar  hadoop-mapreduce-examples-3.2.0.jar         sources
hadoop-mapreduce-client-hs-3.2.0.jar      hadoop-mapreduce-client-nativetask-3.2.0.jar       jdiff
[hadoop@hadoop01 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.2.0.jar
An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
  dbcount: An example job that count the pageview counts from a database.
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A job that counts words from several files.
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
  randomwriter: A map/reduce program that writes 10GB of random data per node.
  secondarysort: An example defining a secondary sort to the reduce.
  sort: A map/reduce program that sorts the data written by the random writer.
  sudoku: A sudoku solver.
  teragen: Generate data for the terasort
  terasort: Run the terasort
  teravalidate: Checking results of terasort
  wordcount: A map/reduce program that counts the words in the input files.
  wordmean: A map/reduce program that counts the average length of the words in the input files.
  wordmedian: A map/reduce program that counts the median length of the words in the input files.
  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
[hadoop@hadoop01 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.2.0.jar pi 10 10
Number of Maps  = 10
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
2019-08-27 13:47:11,866 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.1.100:8032
2019-08-27 13:47:12,179 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1566884685380_0001
2019-08-27 13:47:12,285 INFO input.FileInputFormat: Total input files to process : 10
2019-08-27 13:47:12,341 INFO mapreduce.JobSubmitter: number of splits:10
2019-08-27 13:47:12,372 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2019-08-27 13:47:12,479 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1566884685380_0001
2019-08-27 13:47:12,480 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-08-27 13:47:12,645 INFO conf.Configuration: resource-types.xml not found
2019-08-27 13:47:12,645 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-08-27 13:47:13,018 INFO impl.YarnClientImpl: Submitted application application_1566884685380_0001
2019-08-27 13:47:13,099 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1566884685380_0001/
2019-08-27 13:47:13,099 INFO mapreduce.Job: Running job: job_1566884685380_0001
2019-08-27 13:47:20,205 INFO mapreduce.Job: Job job_1566884685380_0001 running in uber mode : false
2019-08-27 13:47:20,209 INFO mapreduce.Job:  map 0% reduce 0%
2019-08-27 13:47:27,371 INFO mapreduce.Job:  map 20% reduce 0%
2019-08-27 13:47:46,535 INFO mapreduce.Job:  map 20% reduce 7%
2019-08-27 13:47:50,559 INFO mapreduce.Job:  map 40% reduce 7%
2019-08-27 13:47:51,570 INFO mapreduce.Job:  map 50% reduce 7%
2019-08-27 13:47:53,586 INFO mapreduce.Job:  map 60% reduce 7%
2019-08-27 13:47:58,631 INFO mapreduce.Job:  map 60% reduce 20%
2019-08-27 13:47:59,641 INFO mapreduce.Job:  map 80% reduce 20%
2019-08-27 13:48:00,665 INFO mapreduce.Job:  map 100% reduce 20%
2019-08-27 13:48:01,682 INFO mapreduce.Job:  map 100% reduce 100%
2019-08-27 13:48:01,708 INFO mapreduce.Job: Job job_1566884685380_0001 completed successfully
2019-08-27 13:48:01,780 INFO mapreduce.Job: Counters: 54
    File System Counters
        FILE: Number of bytes read=226
        FILE: Number of bytes written=2443397
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=2640
        HDFS: Number of bytes written=215
        HDFS: Number of read operations=45
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=3
        HDFS: Number of bytes read erasure-coded=0
    Job Counters
        Launched map tasks=10
        Launched reduce tasks=1
        Data-local map tasks=10
        Total time spent by all maps in occupied slots (ms)=270199
        Total time spent by all reduces in occupied slots (ms)=31653
        Total time spent by all map tasks (ms)=270199
        Total time spent by all reduce tasks (ms)=31653
        Total vcore-milliseconds taken by all map tasks=270199
        Total vcore-milliseconds taken by all reduce tasks=31653
        Total megabyte-milliseconds taken by all map tasks=276683776
        Total megabyte-milliseconds taken by all reduce tasks=32412672
    Map-Reduce Framework
        Map input records=10
        Map output records=20
        Map output bytes=180
        Map output materialized bytes=280
        Input split bytes=1460
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=280
        Reduce input records=20
        Reduce output records=0
        Spilled Records=40
        Shuffled Maps =10
        Failed Shuffles=0
        Merged Map outputs=10
        GC time elapsed (ms)=67681
        CPU time spent (ms)=63700
        Physical memory (bytes) snapshot=2417147904
        Virtual memory (bytes) snapshot=30882955264
        Total committed heap usage (bytes)=2966421504
        Peak Map Physical memory (bytes)=382750720
        Peak Map Virtual memory (bytes)=2810384384
        Peak Reduce Physical memory (bytes)=181923840
        Peak Reduce Virtual memory (bytes)=2815541248
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=1180
    File Output Format Counters
        Bytes Written=97
Job Finished in 49.977 seconds
Estimated value of Pi is 3.20000000000000000000