Hadoop基础-完全分布式模式部署yarn日志聚集功能

                                        作者:尹正杰

版权声明:原创作品,谢绝转载!否则将追究法律责任。

  其实我们不用配置也可以在服务器后台通过命令行的形式查看相应的日志,但为了更方便查看日志,我们可以将其配置成通过webUI的形式访问日志,本篇博客会手把手的教你如何实操。如果你的集群配置比较低的话,并不建议开启日志,但是一般的大数据集群,服务器配置应该都不低,不过最好根据实际情况考虑。

一.查看日志信息

1>.通过web界面查看日志信息

2>.webUI默认是无法查看到日志信息

3>.通过命令行查看

  日志默认存放在安装hadoop目录的logs文件夹中,其实我们不用配置web页面也可以查看相应的日志,下图就是通过命令行的形式查看日志。

二.配置日志聚集功能

1>.停止hadoop集群

[yinzhengjie@s101 ~]$ stop-dfs.sh
Stopping namenodes on [s101 s105]
s101: stopping namenode
s105: stopping namenode
s103: stopping datanode
s104: stopping datanode
s102: stopping datanode
Stopping journal nodes [s102 s103 s104]
s102: stopping journalnode
s104: stopping journalnode
s103: stopping journalnode
Stopping ZK Failover Controllers on NN hosts [s101 s105]
s101: stopping zkfc
s105: stopping zkfc
[yinzhengjie@s101 ~]$

停止hdfs分布式文件系统([yinzhengjie@s101 ~]$ stop-dfs.sh )

[yinzhengjie@s101 ~]$ stop-yarn.sh
stopping yarn daemons
s101: stopping resourcemanager
s105: no resourcemanager to stop
s102: stopping nodemanager
s104: stopping nodemanager
s103: stopping nodemanager
s102: nodemanager did not stop gracefully after seconds: killing with kill -
s104: nodemanager did not stop gracefully after seconds: killing with kill -
s103: nodemanager did not stop gracefully after seconds: killing with kill -
no proxyserver to stop
[yinzhengjie@s101 ~]$

停止yarn集群([yinzhengjie@s101 ~]$ stop-yarn.sh )

[yinzhengjie@s101 ~]$ mr-jobhistory-daemon.sh stop historyserver
stopping historyserver
[yinzhengjie@s101 ~]$

停止yarn日志服务([yinzhengjie@s101 ~]$ mr-jobhistory-daemon.sh stop historyserver)

[yinzhengjie@s101 ~]$ more `which xcall.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com #判断用户是否传参
if [ $# -lt ];then
echo "请输入参数"
exit
fi #获取用户输入的命令
cmd=$@ for (( i=;i<=;i++ ))
do
#使终端变绿色
tput setaf
echo ============= s$i $cmd ============
#使终端变回原来的颜色,即白灰色
tput setaf
#远程执行命令
ssh s$i $cmd
#判断命令是否执行成功
if [ $? == ];then
echo "命令执行成功"
fi
done
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
Jps
命令执行成功
============= s102 jps ============
Jps
QuorumPeerMain
命令执行成功
============= s103 jps ============
Jps
QuorumPeerMain
命令执行成功
============= s104 jps ============
QuorumPeerMain
Jps
命令执行成功
============= s105 jps ============
Jps
命令执行成功
[yinzhengjie@s101 ~]$

检查是否停止成功([yinzhengjie@s101 ~]$ xcall.sh jps)

2>.修改“yarn-site.xml”配置文件

 [yinzhengjie@s101 ~]$ more /soft/hadoop/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>s101</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property> <property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property> <property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property> <property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property> <!-- 日志保留时间设置7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property> </configuration> <!-- yarn-site.xml配置文件的作用:
#主要用于配置调度器级别的参数. yarn.resourcemanager.hostname 参数的作用:
#指定资源管理器(resourcemanager)的主机名 yarn.nodemanager.aux-services 参数的作用:
#指定nodemanager使用shuffle yarn.nodemanager.pmem-check-enabled 参数的作用:
#是否启动一个线程检查每个任务正使用的物理内存量,如果任务超出分配值,则直接将其杀掉,默认是true yarn.nodemanager.vmem-check-enabled 参数的作用:
#是否启动一个线程检查每个任务正使用的虚拟内存量,如果任务超出分配值,则直接将其杀掉,默认是true yarn.log-aggregation-enable 参数的作用:
#是否开启webUI日志聚集功能使能,默认为flase yarn.log-aggregation.retain-seconds 参数的作用:
#指定日志保留时间,单位为妙
-->
[yinzhengjie@s101 ~]$

3>.分发配置文件到各个节点

[yinzhengjie@s101 ~]$ more `which xrsync.sh`
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com #判断用户是否传参
if [ $# -lt ];then
echo "请输入参数";
exit
fi #获取文件路径
file=$@ #获取子路径
filename=`basename $file` #获取父路径
dirpath=`dirname $file` #获取完整路径
cd $dirpath
fullpath=`pwd -P` #同步文件到DataNode
for (( i=;i<=;i++ ))
do
#使终端变绿色
tput setaf
echo =========== s$i %file ===========
#使终端变回原来的颜色,即白灰色
tput setaf
#远程执行命令
rsync -lr $filename `whoami`@s$i:$fullpath
#判断命令是否执行成功
if [ $? == ];then
echo "命令执行成功"
fi
done
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ xrsync.sh /soft/hadoop-2.7./
=========== s102 %file ===========
命令执行成功
=========== s103 %file ===========
命令执行成功
=========== s104 %file ===========
命令执行成功
=========== s105 %file ===========
命令执行成功
[yinzhengjie@s101 ~]$

4>.启动hadoop集群

[yinzhengjie@s101 ~]$ start-dfs.sh
Starting namenodes on [s101 s105]
s105: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s105.out
s101: starting namenode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-namenode-s101.out
s103: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s103.out
s102: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s102.out
s104: starting datanode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-datanode-s104.out
Starting journal nodes [s102 s103 s104]
s102: starting journalnode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-journalnode-s102.out
s103: starting journalnode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-journalnode-s103.out
s104: starting journalnode, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-journalnode-s104.out
Starting ZK Failover Controllers on NN hosts [s101 s105]
s105: starting zkfc, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-zkfc-s105.out
s101: starting zkfc, logging to /soft/hadoop-2.7./logs/hadoop-yinzhengjie-zkfc-s101.out
[yinzhengjie@s101 ~]$

启动hdfs分布式文件系统([yinzhengjie@s101 ~]$ start-dfs.sh )

[yinzhengjie@s101 ~]$ start-yarn.sh
starting yarn daemons
s101: starting resourcemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-resourcemanager-s101.out
s105: starting resourcemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-resourcemanager-s105.out
s102: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s102.out
s103: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s103.out
s104: starting nodemanager, logging to /soft/hadoop-2.7./logs/yarn-yinzhengjie-nodemanager-s104.out
[yinzhengjie@s101 ~]$

启动yarn集群([yinzhengjie@s101 ~]$ start-yarn.sh )

[yinzhengjie@s101 ~]$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /soft/hadoop-2.7./logs/mapred-yinzhengjie-historyserver-s101.out
[yinzhengjie@s101 ~]$

启动yarn日志服务([yinzhengjie@s101 ~]$ mr-jobhistory-daemon.sh start historyserver)

[yinzhengjie@s101 ~]$ xcall.sh jps
============= s101 jps ============
DFSZKFailoverController
Jps
NameNode
JobHistoryServer
ResourceManager
命令执行成功
============= s102 jps ============
DataNode
JournalNode
NodeManager
Jps
QuorumPeerMain
命令执行成功
============= s103 jps ============
JournalNode
NodeManager
Jps
DataNode
QuorumPeerMain
命令执行成功
============= s104 jps ============
JournalNode
NodeManager
QuorumPeerMain
Jps
DataNode
命令执行成功
============= s105 jps ============
NameNode
Jps
DFSZKFailoverController
命令执行成功
[yinzhengjie@s101 ~]$

检查集群是否启动成功([yinzhengjie@s101 ~]$ xcall.sh jps)

5>.在yarn上执行MapReduce程序

[yinzhengjie@s101 ~]$ hdfs dfs -rm -R /yinzhengjie/data/output
// :: INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = minutes, Emptier interval = minutes.
Deleted /yinzhengjie/data/output
[yinzhengjie@s101 ~]$
[yinzhengjie@s101 ~]$ hadoop jar /soft/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7..jar wordcount /yinzhengjie/data/input /yinzhengjie/data/output
// :: INFO client.RMProxy: Connecting to ResourceManager at s101/172.30.1.101:
// :: INFO input.FileInputFormat: Total input paths to process :
// :: INFO mapreduce.JobSubmitter: number of splits:
// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1534857666985_0001
// :: INFO impl.YarnClientImpl: Submitted application application_1534857666985_0001
// :: INFO mapreduce.Job: The url to track the job: http://s101:8088/proxy/application_1534857666985_0001/
// :: INFO mapreduce.Job: Running job: job_1534857666985_0001
// :: INFO mapreduce.Job: Job job_1534857666985_0001 running in uber mode : false
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: map % reduce %
// :: INFO mapreduce.Job: Job job_1534857666985_0001 completed successfully
// :: INFO mapreduce.Job: Counters:
File System Counters
FILE: Number of bytes read=
FILE: Number of bytes written=
FILE: Number of read operations=
FILE: Number of large read operations=
FILE: Number of write operations=
HDFS: Number of bytes read=
HDFS: Number of bytes written=
HDFS: Number of read operations=
HDFS: Number of large read operations=
HDFS: Number of write operations=
Job Counters
Launched map tasks=
Launched reduce tasks=
Data-local map tasks=
Total time spent by all maps in occupied slots (ms)=
Total time spent by all reduces in occupied slots (ms)=
Total time spent by all map tasks (ms)=
Total time spent by all reduce tasks (ms)=
Total vcore-milliseconds taken by all map tasks=
Total vcore-milliseconds taken by all reduce tasks=
Total megabyte-milliseconds taken by all map tasks=
Total megabyte-milliseconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=
Map output records=
Map output bytes=
Map output materialized bytes=
Input split bytes=
Combine input records=
Combine output records=
Reduce input groups=
Reduce shuffle bytes=
Reduce input records=
Reduce output records=
Spilled Records=
Shuffled Maps =
Failed Shuffles=
Merged Map outputs=
GC time elapsed (ms)=
CPU time spent (ms)=
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
BAD_ID=
CONNECTION=
IO_ERROR=
WRONG_LENGTH=
WRONG_MAP=
WRONG_REDUCE=
File Input Format Counters
Bytes Read=
File Output Format Counters
Bytes Written=
[yinzhengjie@s101 ~]$

6>.查看yarn的记录信息

7>.查看历史日志

8>.查看日志信息

   

 

Hadoop基础-完全分布式模式部署yarn日志聚集功能的更多相关文章

  1. 启用yarn日志聚集功能

    在yarn-site.xml配置文件中添加如下内容: ##开启日志聚集功能        <property>                <name>yarn.log-ag ...

  2. Hadoop基础-HDFS分布式文件系统的存储

    Hadoop基础-HDFS分布式文件系统的存储 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.HDFS数据块 1>.磁盘中的数据块 每个磁盘都有默认的数据块大小,这个磁盘 ...

  3. Yarn 的日志聚集功能配置使用

    需要  hadoop 的安装目录/etc/hadoop/yarn-site.xml 中进行配置 配置内容 <property> <name>yarn.log-aggregati ...

  4. hadoop 3.x 配置日志聚集功能

    打开$HADOOP_HOME/etc/hadoop/yarn-site.xml,增加以下配置(在此配置文件中尽量不要使用中文注释) <!--logs--> <property> ...

  5. 开启spark日志聚集功能

    spark监控应用方式: 1)在运行过程中可以通过web Ui:4040端口进行监控 2)任务运行完成想要监控spark,需要启动日志聚集功能 开启日志聚集功能方法: 编辑conf/spark-env ...

  6. Hadoop伪分布式模式部署

    Hadoop的安装有三种执行模式: 单机模式(Local (Standalone) Mode):Hadoop的默认模式,0配置.Hadoop执行在一个Java进程中.使用本地文件系统.不使用HDFS, ...

  7. 用python + hadoop streaming 编写分布式程序(三) -- 自定义功能

    又是期末又是实训TA的事耽搁了好久……先把写好的放上博客吧 相关随笔: Hadoop-1.0.4集群搭建笔记 用python + hadoop streaming 编写分布式程序(一) -- 原理介绍 ...

  8. Hadoop+Hbas完全分布式安装部署

    Hadoop安装部署基本步骤: 1.安装jdk,配置环境变量. jdk可以去网上自行下载,环境变量如下: 编辑  vim  /etc/profile 文件,添加如下内容: export JAVA_HO ...

  9. hadoop实战之分布式模式

    环境 192.168.1.101 host101 192.168.1.102 host102 1.安装配置host101 [root@host101 ~]# cat /etc/hosts |grep ...

随机推荐

  1. 第一次Spring总结

    第一阶段:下载了类似app使用,并做了对比,分析,对自己的app有了一些构思,完成了环境的配置.在这一阶段,一开始只有两个女生显得有点弱,面对从未接触过的app项目,首先就是配置环境方面的,在经过班上 ...

  2. JavaScript浏览器历史的语法小问题

    https://www.w3schools.com/jsref/met_his_back.asp This is the same as clicking the "Back button& ...

  3. postman发送json格式的post请求

    在地址栏里输入请求url:http://127.0.0.1:8081/getmoney 选择“POST”方式, 在“headers”添加key:Content-Type  , value:applic ...

  4. Linux大页内存管理等---菜鸟初学

    1. 查看linux的内存情况: free -m 2. 查看是否开启大页的方法: cat /proc/meminfo |grep -i HugePage AnonHugePages: 276480 k ...

  5. spark中saveAsTextFile的错误

    写了很简单的一段spark代码,将结果保存为windows本地文件,执行之后总是报错NullPointerException 查询之后 发现是本地缺少hadoop需要的一个文件所致 如果本地已经安装了 ...

  6. SVN上线步骤笔记

    项目代码位置: /data/svn/play_out 项目代码目录名称: test SVN创建位置:/data/svn/repos_Websvn线上地址:svn://192.168.1.1/repos ...

  7. XML的基礎結構

    1.xml是什麼? xml,Extensible Markup Language,扩展性标识语言,後綴名為.xml. 2.xml有什麼功能? xml功能是傳輸和儲存數據,用於不同的應用和平台數據共享和 ...

  8. CSS实现水平居中的5种思路

    前面的话 水平居中是经常遇到的问题.看似方法较多,条条大路通罗马.但系统梳理下,其实都围绕着几个思路展开.本文将介绍关于水平居中的5种思路 text-align [思路一]:在父元素中设置text-a ...

  9. JAVA中接口与抽象类

    抽象类总结 抽象类的概念:是使用关键字abstract修饰的类就是抽象类: 抽象类的产生:当多个不能相互继承的类具有相同的功能时,就需要将共同的信息向上抽取,放到公共的父类中:如果公共的父类只能描述所 ...

  10. Pairs Forming LCM LightOJ - 1236 (算术基本定理)

    题意: 就是求1-n中有多少对i 和 j 的最小公倍数为n  (i <= j) 解析: 而这题,我们假设( a , b ) = n ,那么: n=pk11pk22⋯pkss, a=pd11pd2 ...