Cloudera Certified Associate Administrator案例之Troubleshoot

                                      作者:尹正杰

版权声明:原创作品,谢绝转载!否则将追究法律责任。

一.调整日志的进程级别

问题描述:
  今天node103.yinzhengjie.org.cn节点的DataNode和NodeManager进程频繁死掉,你决定临时将该节点两个进程的日志级别调整为DEBUG,以便于进行维护排查。 解决方案:
  操作时需注意要求将HDFS和YARN的所有节点日志界别都调整,还是只需要调整还是只需要调整个别节点,两者的调整方式是不一样的,对于个别节点日志级别的临时调整,可以使用yarn daemonlog -setlevel命令来进行,当然通过CM进行更为方便。

1>.点击主机,选择"所有主机"

2>.点击"HDFS DataNode"角色

3>.进入"node103.yinzhengjie.org.cn"节点的DataNode角色的配置界面(需要注意的是,此配置只针对该节点生效哟~)

4>.点击配置搜索关键字"DataNode 记录阈值"(英文页面为:"DataNode Logging Threshold")

5>.修改"node103.yinzhengjie.org.cn"节点的页面配置为"DEBUG"

6>.进入NodeManger的WebUI配置界面

7>.点击配置搜索关键字"NodeManager 记录阈值"(英文页面为:"NodeManager Logging Threshold")

二.运行wordcount

问题描述:
  公司的某个开发人员尝试在集群上运行wordcount程序,但作业执行发生错误,请你帮助解决。
  请将gateway机器的input.txxt文件上传到HDFS上的“/yinzhengjie/debug/mapreduce/data/input”目录中,并执行"wordcount /yinzhengjie/debug/mapreduce/data/input /yinzhengjie/debug/mapreduce/data/output"来测试是否可以运行。 解决方案:
  通常作业失败类型只有目录存在,内存不足,权限部队等简单的几种,难度较低,一般不需要Java语言的知识,只要能看懂英文就可以解决。

1>.上传文件

[root@node101.yinzhengjie.org.cn ~]# ll
total
-rw-r--r-- root root Jun : input.txt
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# cat input.txt
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -mkdir -p /yinzhengjie/debug/mapreduce/data/input
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -copyFromLocal input.txt /yinzhengjie/debug/mapreduce/data/input
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -ls /yinzhengjie/debug/mapreduce/data/input
Found items
-rw-r--r-- root supergroup -- : /yinzhengjie/debug/mapreduce/data/input/input.txt
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]#

2>.执行wordcount

[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# su hdfs
[hdfs@node101.yinzhengjie.org.cn /root]$
[hdfs@node101.yinzhengjie.org.cn /root]$ cd /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ hadoop jar hadoop-mapreduce-examples.jar wordcount /yinzhengjie/debug/mapreduce/data/input /yinzhengjie/debug/mapreduce/data/output
// :: INFO client.RMProxy: Connecting to ResourceManager at node101.yinzhengjie.org.cn/172.30.1.101:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560591816714_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1560591816714_0001
// :: INFO mapreduce.Job: The url to track the job: http://node101.yinzhengjie.org.cn:8088/proxy/application_1560591816714_0001/
// :: INFO mapreduce.Job: Running job: job_1560591816714_0001
// :: INFO mapreduce.Job: Job job_1560591816714_0001 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1560591816714_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Launched reduce tasks=
Data-local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total time spent by all reduce tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total vcore-milliseconds taken by all reduce tasks=
Total megabyte-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=
Map output records=
Map output bytes=
Map output materialized bytes=
Input split bytes=
Combine input records=
Combine output records=
Reduce input groups=
Reduce shuffle bytes=
Reduce input records=
Reduce output records=
Spilled Records=
Shuffled Maps =
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
BAD_ID=
CONNECTION=
IO_ERROR=
WRONG_LENGTH=
WRONG_MAP=
WRONG_REDUCE=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$

[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ hadoop jar hadoop-mapreduce-examples.jar wordcount /yinzhengjie/debug/mapreduce/data/input /yinzhengjie/debug/mapreduce/data/output

三.

Cloudera Certified Associate Administrator案例之Troubleshoot篇的更多相关文章

  1. Cloudera Certified Associate Administrator案例之Test篇

    Cloudera Certified Associate Administrator案例之Test篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.准备工作(将CM升级到&qu ...

  2. Cloudera Certified Associate Administrator案例之Manage篇

    Cloudera Certified Associate Administrator案例之Manage篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.下载Namenode镜像 ...

  3. Cloudera Certified Associate Administrator案例之Install篇

    Cloudera Certified Associate Administrator案例之Install篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.创建主机模板(为了给主 ...

  4. Cloudera Certified Associate Administrator案例之Configure篇

    Cloudera Certified Associate Administrator案例之Configure篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.下载CDH集群中最 ...

  5. Flume实战案例运维篇

    Flume实战案例运维篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.Flume概述 1>.什么是Flume Flume是一个分布式.可靠.高可用的海量日志聚合系统,支 ...

  6. CNCF基金会的Certified Kubernetes Administrator认证考试计划

    关于CKA考试 CKA(Certified Kubernetes Administrator)是CNCF基金会(Cloud Native Computing Foundation)官方推出的Kuber ...

  7. 分享数百个 HT 工业互联网 2D 3D 可视化应用案例之 2019 篇

    继<分享数百个 HT 工业互联网 2D 3D 可视化应用案例>2018 篇,图扑软件定义 2018 为国内工业互联网可视化的元年后,2019 年里我们与各行业客户进行了更深度合作,拓展了H ...

  8. 数百个 HT 工业互联网 2D 3D 可视化应用案例分享 - 2019 篇

    继<分享数百个 HT 工业互联网 2D 3D 可视化应用案例>2018 篇,图扑软件定义 2018 为国内工业互联网可视化的元年后,2019 年里我们与各行业客户进行了更深度合作,拓展了H ...

  9. robotframework+selenium搭配chrome浏览器,web测试案例(搭建篇)

    这两天发布版本 做的事情有点多,都没有时间努力学习了,先给自己个差评,今天折腾了一天, 把robotframework 和 selenium 还有appnium 都研究了一下 ,大概有个谱,先说说we ...

随机推荐

  1. maven-archetype-plugin 的正确打开方式

    1.  准备好一个编辑好的模板工程 2. 在 pom.xml 中添加 maven-archetype-plugin 插件 <plugin> <groupId>org.apach ...

  2. Multi-Channel Buffers

    This describes a 4 channels buffer of 16 bit samples.Data organisation :Sample 1, front left speaker ...

  3. 【ARM-Linux开发】Linux模块机制浅析

    Linux模块机制浅析   Linux允许用户通过插入模块,实现干预内核的目的.一直以来,对linux的模块机制都不够清晰,因此本文对内核模块的加载机制进行简单地分析. 模块的Hello World! ...

  4. ReetrantLock架构源码 --- One

    以下是绅士通过processon画的一个比较简单的架构,模板模式理清楚确实需要一点点时间 Doug Lea牛ban- .- 最近在复习整理知识点,这上面的一些关键方法addWaiter();acqui ...

  5. react中的ref在input中的详解

    当我们在项目中遇见文本输入框的时候,获取时刻输入框中的值 1.受控组件 class NameForm extends React.Component { constructor(props) { su ...

  6. 长乐国庆集训Day3

    T1 动态逆序对 题目 [题目描述] 给出一个长度为n的排列a(1~n这n个数在数列中各出现1次).每次交换两个数,求逆序对数%2的结果. 逆序对:对于两个数a[i],a[j](i<j),若a[ ...

  7. a++与++a,谈谈C++的参数传递

    先看一段代码: #include<iostream> using namespace std; void func(int a, int b) { cout << a < ...

  8. [python 2.x] xml.etree.ElementTree module

    XML 文件:xmlparse.xml <?xml version="1.0" encoding="UTF-8" standalone="no& ...

  9. PB 获取或操作数据窗口语句的方法

    1.setsqlselect用法: ls_select=getsqlselect    //通过getsqlselect取得当前数据窗口的查询语句 ls_where="  "    ...

  10. 洛谷 P1411 树 (树形dp)

    大意: 给定树, 求删除一些边, 使得连通块大小的乘积最大 设$dp_{i,j}$表示只考虑点$i$的子树, $i$所在连通块大小为$j$的最大值. 转移的时候不计算$i$所在连通块的贡献, 留到最后 ...