Cloudera Certified Associate Administrator案例之Troubleshoot篇

　　　　　　Cloudera Certified Associate Administrator案例之Troubleshoot篇

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.调整日志的进程级别

问题描述：
　　今天node103.yinzhengjie.org.cn节点的DataNode和NodeManager进程频繁死掉，你决定临时将该节点两个进程的日志级别调整为DEBUG，以便于进行维护排查。

解决方案：
　　操作时需注意要求将HDFS和YARN的所有节点日志界别都调整，还是只需要调整还是只需要调整个别节点，两者的调整方式是不一样的，对于个别节点日志级别的临时调整，可以使用yarn daemonlog -setlevel命令来进行，当然通过CM进行更为方便。

1>.点击主机，选择"所有主机"

2>.点击"HDFS DataNode"角色

3>.进入"node103.yinzhengjie.org.cn"节点的DataNode角色的配置界面（需要注意的是，此配置只针对该节点生效哟~）

4>.点击配置搜索关键字"DataNode 记录阈值"(英文页面为:"DataNode Logging Threshold")

5>.修改"node103.yinzhengjie.org.cn"节点的页面配置为"DEBUG"

6>.进入NodeManger的WebUI配置界面

7>.点击配置搜索关键字"NodeManager 记录阈值"(英文页面为:"NodeManager Logging Threshold")

二.运行wordcount

问题描述：
　　公司的某个开发人员尝试在集群上运行wordcount程序，但作业执行发生错误，请你帮助解决。
　　请将gateway机器的input.txxt文件上传到HDFS上的“/yinzhengjie/debug/mapreduce/data/input”目录中，并执行"wordcount /yinzhengjie/debug/mapreduce/data/input /yinzhengjie/debug/mapreduce/data/output"来测试是否可以运行。

解决方案：
　　通常作业失败类型只有目录存在，内存不足，权限部队等简单的几种，难度较低，一般不需要Java语言的知识，只要能看懂英文就可以解决。

1>.上传文件

[root@node101.yinzhengjie.org.cn ~]# ll
total
-rw-r--r--  root root  Jun  : input.txt
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# cat input.txt
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -mkdir -p /yinzhengjie/debug/mapreduce/data/input
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -copyFromLocal input.txt /yinzhengjie/debug/mapreduce/data/input
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -ls /yinzhengjie/debug/mapreduce/data/input
Found  items
-rw-r--r--    root supergroup         -- : /yinzhengjie/debug/mapreduce/data/input/input.txt
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# 

2>.执行wordcount

[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# su hdfs
[hdfs@node101.yinzhengjie.org.cn /root]$
[hdfs@node101.yinzhengjie.org.cn /root]$ cd /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ hadoop jar hadoop-mapreduce-examples.jar wordcount /yinzhengjie/debug/mapreduce/data/input  /yinzhengjie/debug/mapreduce/data/output
// :: INFO client.RMProxy: Connecting to ResourceManager at node101.yinzhengjie.org.cn/172.30.1.101:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560591816714_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1560591816714_0001
// :: INFO mapreduce.Job: The url to track the job: http://node101.yinzhengjie.org.cn:8088/proxy/application_1560591816714_0001/
// :: INFO mapreduce.Job: Running job: job_1560591816714_0001
// :: INFO mapreduce.Job: Job job_1560591816714_0001 running in uber mode : false
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Job job_1560591816714_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
    File System Counters
        FILE: Number of bytes read=
        FILE: Number of bytes written=
        FILE: Number of read operations=
        FILE: Number of large read operations=
        FILE: Number of write operations=
        HDFS: Number of bytes read=
        HDFS: Number of bytes written=
        HDFS: Number of read operations=
        HDFS: Number of large read operations=
        HDFS: Number of write operations=
    Job Counters
        Launched map tasks=
        Launched reduce tasks=
        Data-local map tasks=
        Total time spent by all maps in occupied slots (ms)=
        Total time spent by all reduces in occupied slots (ms)=
        Total time spent by all map tasks (ms)=
        Total time spent by all reduce tasks (ms)=
        Total vcore-milliseconds taken by all map tasks=
        Total vcore-milliseconds taken by all reduce tasks=
        Total megabyte-milliseconds taken by all map tasks=
        Total megabyte-milliseconds taken by all reduce tasks=
    Map-Reduce Framework
        Map input records=
        Map output records=
        Map output bytes=
        Map output materialized bytes=
        Input split bytes=
        Combine input records=
        Combine output records=
        Reduce input groups=
        Reduce shuffle bytes=
        Reduce input records=
        Reduce output records=
        Spilled Records=
        Shuffled Maps =
        Failed Shuffles=
        Merged Map outputs=
        GC time elapsed (ms)=
        CPU time spent (ms)=
        Physical memory (bytes) snapshot=
        Virtual memory (bytes) snapshot=
        Total committed heap usage (bytes)=
    Shuffle Errors
        BAD_ID=
        CONNECTION=
        IO_ERROR=
        WRONG_LENGTH=
        WRONG_MAP=
        WRONG_REDUCE=
    File Input Format Counters
        Bytes Read=
    File Output Format Counters
        Bytes Written=
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ 

[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ hadoop jar hadoop-mapreduce-examples.jar wordcount /yinzhengjie/debug/mapreduce/data/input /yinzhengjie/debug/mapreduce/data/output

三.