Ubutnu 20.04 安装和使用单机版hadoop 3.2 [转载]

• 按照此文档操作，可以一次部署成功：Ubutnu 20.04 安装和使用单机版hadoop 3.2

• 部署之后，提交测试任务报资源问题。原因是yarn还需要配置，如下：

$ cat yarn-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
        <!-- 指定MR走shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <!-- 指定ResourceManager的地址-->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>192.168.122.200</value>
    </property>

    <!-- 环境变量的继承 -->
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>

</configuration>

注意，一开始的配置中包含有资源限制方面的配置，但是会导致map任务一直卡在0%，所以将那些资源限制删除了。

• 示例：

$ hadoop jar ~/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar wordcount /1 /2 /out3
2023-01-06 10:23:58,058 INFO client.RMProxy: Connecting to ResourceManager at /192.168.122.200:8032
2023-01-06 10:23:58,362 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1672999900900_0002
2023-01-06 10:23:58,508 INFO input.FileInputFormat: Total input files to process : 2
2023-01-06 10:23:58,557 INFO mapreduce.JobSubmitter: number of splits:2
2023-01-06 10:23:58,701 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1672999900900_0002
2023-01-06 10:23:58,702 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-01-06 10:23:58,796 INFO conf.Configuration: resource-types.xml not found
2023-01-06 10:23:58,796 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-01-06 10:23:58,829 INFO impl.YarnClientImpl: Submitted application application_1672999900900_0002
2023-01-06 10:23:58,849 INFO mapreduce.Job: The url to track the job: http://192.168.122.200:8088/proxy/application_1672999900900_0002/
2023-01-06 10:23:58,849 INFO mapreduce.Job: Running job: job_1672999900900_0002
2023-01-06 10:24:03,962 INFO mapreduce.Job: Job job_1672999900900_0002 running in uber mode : false
2023-01-06 10:24:03,965 INFO mapreduce.Job:  map 0% reduce 0%
2023-01-06 10:24:08,059 INFO mapreduce.Job:  map 100% reduce 0%
2023-01-06 10:24:11,079 INFO mapreduce.Job:  map 100% reduce 100%
2023-01-06 10:24:12,116 INFO mapreduce.Job: Job job_1672999900900_0002 completed successfully
...