问题背景

本次进行一个项目的重构,在某些活动数据量比较大的情况下,会偶尔出现1200s超时的情况,如下:

AttemptID:attempt_1410771599055_11709_m_000033_0 Timed out after 1200 secs

而hadoop会不断启动备份任务进行重试,重试也许成功,但失败的概率还是比较大:

经过分析,hadoop的任务都有个超时时间,使用下面的参数设置,表示1200s后如果没有进展,就会任务该任务超时,将其状态设置为FAILED。

-Dmapreduce.task.timeout=1200000

到底因为什么原因导致超时?为了继续分析这个问题,我们将这个参数设置得非常之大。

调整超时参数

将超时时间设置为24小时之后,发现任务不会FAILED,但是其执行了大概40多个小时,仍然还没有执行完成。

还好我们在任务执行过程中打了不少的log,以帮助分析问题。经过日志的分析,我们发现有下面的现象:

2014-09-22 00:17:29,005 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 77477) is collected!
2014-09-22 00:17:29,005 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 77477) is collected!
2014-09-22 00:17:29,005 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 77477) is collected!
2014-09-22 01:17:29,054 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 120096) is collected!
2014-09-22 01:17:29,064 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 120096) is collected!
2014-09-22 01:17:29,064 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 120096) is collected!
2014-09-22 01:17:29,064 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 120096) is collected!
...
2014-09-22 01:17:36,590 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 164747) is collected!
2014-09-22 01:17:36,590 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 164747) is collected!
2014-09-22 01:17:36,590 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 164747) is collected!
2014-09-22 01:17:36,590 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 164747) is collected!
2014-09-22 02:17:36,674 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 158198) is collected!
2014-09-22 02:17:36,683 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 158198) is collected!
2014-09-22 02:17:36,683 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 158198) is collected!
2014-09-22 02:17:36,683 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 158198) is collected!
....
2014-09-22 02:17:40,888 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 203233) is collected!
2014-09-22 02:17:40,888 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 203233) is collected!
2014-09-22 02:17:40,888 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 203233) is collected!
2014-09-22 03:17:40,925 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 79188) is collected!
2014-09-22 03:17:40,934 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 79188) is collected!
2014-09-22 03:17:40,934 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 79188) is collected!
2014-09-22 03:17:40,934 INFO [main] com.xxx.yo.phase1.Phase1Mapper: history(caid: 2000037, superid: 79188) is collected!

日志分析得出的结论便是,程序总会在某个时间点休息3600秒(大概1个小时),然后再执行一会儿,便又休息3600秒。

hadoop configuration中得出初步结论

我们在这1个小时对该java进程进行监控,发现该进程在此期间(jstack命令查看其日志),一直在一个点等待:

"main" prio=10 tid=0x000000000293f000 nid=0x1e06 runnable [0x0000000041b20000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006e243c3f0> (a sun.nio.ch.Util$2)
- locked <0x00000006e243c3e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006e243c1a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.readChannelFully(PacketReceiver.java:258)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:209)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:171)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:102)
at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:170)
at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:135)
- locked <0x00000006e12dcc78> (a org.apache.hadoop.hdfs.RemoteBlockReader2)
at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:698)
- eliminated <0x00000006e12dcc18> (a org.apache.hadoop.hdfs.DFSInputStream)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:752)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
- locked <0x00000006e12dcc18> (a org.apache.hadoop.hdfs.DFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at com.xxx.app.MzSequenceFile$PartInputStream.read(MzSequenceFile.java:451)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at org.apache.hadoop.io.Text.readFields(Text.java:292)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at com.xxx.app.MzSequenceFile$Reader.deserializeValue(MzSequenceFile.java:672)
at com.xxx.app.MzSequenceFile$Reader.next(MzSequenceFile.java:684)
at com.xxx.app.MzSequenceFile$Reader.next(MzSequenceFile.java:692)
at com.xxx.yo.io.CombineFileRawLogReader.streamNext(CombineFileRawLogReader.java:284)
at com.xxx.yo.io.CombineFileRawLogReader.next(CombineFileRawLogReader.java:342)
at com.xxx.yo.io.CampaignRawLogReader.next(CampaignRawLogReader.java:73)
at com.xxx.yo.io.CampaignRawLogReader.next(CampaignRawLogReader.java:23)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:197)
- locked <0x00000006e01dd3e0> (a org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:183)
- locked <0x00000006e01dd3e0> (a org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

当时初步怀疑是JDK版本问题,Java NIO也确实存在着臭名昭著的epoll空轮询导致CPU100%问题,但这个bug在JDK6中的高版本就已经解决了,更何况我们使用的是1.7。而且我们通过top -p 进程id的方式查看其CPU占用率为0,排除了该bug。

经过日志的初步分析,发现3600s这个线索,从job的configuration中,初步查找出参数dfs.client.socket-timeout,单位毫秒。

-Ddfs.client.socket-timeout=3600000

试验性地将这个参数修改为60ms,可以看出出现超时的概率非常大,但会不断重试以继续:

2014-09-26 12:53:03,184 WARN [main] org.apache.hadoop.hdfs.DFSClient: Failed to connect to /192.168.7.22:50010 for block, add to deadNodes and continue.
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.7.17:22051 remote=/192.168.7.22:50010]
java.net.SocketTimeoutException: 60 millis timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/192.168.7.17:22051 remote=/192.168.7.22:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1490)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:392)
at org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:131)
at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1108)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:601)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at com.xxx.app.MzSequenceFile$Reader.init(MzSequenceFile.java:521)
at com.xxx.app.MzSequenceFile$Reader.<init>(MzSequenceFile.java:515)
at com.xxx.app.MzSequenceFile$Reader.<init>(MzSequenceFile.java:505)
at com.xxx.yo.io.CombineFileRawLogReader.<init>(CombineFileRawLogReader.java:146)
at com.xxx.yo.io.CampaignRawLogReader.next(CampaignRawLogReader.java:64)
at com.xxx.yo.io.CampaignRawLogReader.next(CampaignRawLogReader.java:22)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:183)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

于是,最终将这个参数设置为60s,这其实也是集群最终默认的超时时间,由于之前一次不明就里的优化,导致了后续这些问题的发生,因此在调整参数时,一定要注意了解清楚该参数造成的影响。

简要分析的结论

在Mapper端读取HDFS上的文件时,可能由于网络原因(由于我们的Split切分地比较大,因此不可能做到完全数据本地化)导致读取数据超时,原来居然设置成1个小时,而任务的超时时间仅设置为20分钟,因此只要发生读取数据超时,就必然会引起任务超时。

通过这次分析过程,学到了很多查找问题的方式,包括通过现象分析规律,得到线索,最终查找问题的原因。快速测试,不能忽略哪怕一个小的Exception,不行就是分析hadoop的源码,掌握其运行时行为。

对于悬挂的task,如果task tracker在一段时间(默认是10min,可以通过mapred.task.timeout属性值来设置,单位是毫秒)内一直没有收到它的进度报告,则把它标记为失效,这时候就会出现前面所说的超时错误。

TaskTracker通过心跳包告知JobTracker某个task attempt失败了,则JobTracker把该task尽量分配给另外一个task tracker来执行。如果同一个task连续4次(该值可以通过mapred.map.max.attempts和mapred.reduce.max.attempts属性值来设置)都执行失败,那JobTracker就不会再做更多的尝试了,本次job也就宣告失败了。

hadoop中的一次集群任务执行超时问题查找过程的更多相关文章

  1. 菜鸟玩云计算之十九:Hadoop 2.5.0 HA 集群安装第2章

    菜鸟玩云计算之十九:Hadoop 2.5.0 HA 集群安装第2章 cheungmine, 2014-10-26 在上一章中,我们准备好了计算机和软件.本章开始部署hadoop 高可用集群. 2 部署 ...

  2. 菜鸟玩云计算之十八:Hadoop 2.5.0 HA 集群安装第1章

    菜鸟玩云计算之十八:Hadoop 2.5.0 HA 集群安装第1章 cheungmine, 2014-10-25 0 引言 在生产环境上安装Hadoop高可用集群一直是一个需要极度耐心和体力的细致工作 ...

  3. Apache Hadoop 2.9.2 的集群管理之服役和退役

    Apache Hadoop 2.9.2 的集群管理之服役和退役 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 随着公司业务的发展,客户量越来越多,产生的日志自然也就越来越大来,可能 ...

  4. Hadoop ha CDH5.15.1-hadoop集群启动后,两个namenode都是standby模式

    Hadoop ha CDH5.15.1-hadoop集群启动后,两个namenode都是standby模式 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一说起周五,想必大家都特别 ...

  5. Hadoop 2.7.3 分布式集群安装

    1. 集群规划: 192.168.1.252 palo252 Namenode+Datanode 192.168.1.253 palo253 YarnManager+Datanode+Secondar ...

  6. Hadoop 系列(二)—— 集群资源管理器 YARN

    一.hadoop yarn 简介 Apache YARN (Yet Another Resource Negotiator) 是 hadoop 2.0 引入的集群资源管理系统.用户可以将各种服务框架部 ...

  7. Hadoop ha CDH5.15.1-hadoop集群启动后,集群容量不正确,莫慌,这是正常的表现!

    Hadoop ha CDH5.15.1-hadoop集群启动后,集群容量不正确,莫慌,这是正常的表现! 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.集群启动顺序 1>. ...

  8. 马士兵hadoop第二课:hdfs集群集中管理和hadoop文件操作

    马士兵hadoop第一课:虚拟机搭建和安装hadoop及启动 马士兵hadoop第二课:hdfs集群集中管理和hadoop文件操作 马士兵hadoop第三课:java开发hdfs 马士兵hadoop第 ...

  9. 马士兵hadoop第二课:hdfs集群集中管理和hadoop文件操作(转)

    马士兵hadoop第一课:虚拟机搭建和安装hadoop及启动 马士兵hadoop第二课:hdfs集群集中管理和hadoop文件操作 马士兵hadoop第三课:java开发hdfs 马士兵hadoop第 ...

随机推荐

  1. CUDA概述

    基本概念和定义 CUDA实际上是一种多平台计算系统,最基本的配置是一台CPU主机和一块显卡,两者都能进行计算.其中CPU主机称为host, GPU称为device,CUDA的API程序的命名和参数命名 ...

  2. SpringInAction--Bean的作用域

    Spring定义了多种作用域,我们在使用的时候可以根据使用的需求来选择对应的作用域,这些作用域,包括(第二个括号中为更安全的注解方法,具体更多参数可查看接口代码) 单例(Singleton)(Conf ...

  3. 添加operations模块

    添加operations模块,models内容为: from django.db import models from datetime import datetime from users.mode ...

  4. 看完前任三,想起我的前任java女程序员,她曾教会我……

    前任三最近非常火了,票房蹭蹭往上升,昨天也去电影院看了,想起了我的前任,她是一名女程序员,为了让我学好java,她曾经亲自教我Java的算法,学算法是件非常重要的事,在这忍住回忆的悲伤,分享给你们. ...

  5. EP-N8530S USB WIFI 驱动移植

    /*********************************************************************** * EP-N8530S USB WIFI 驱动移植 * ...

  6. Hibernate 一对一 (one-to-one)

    一对一(one-to-one)实例(Person-IdCard) 一对一的关系在数据库中表示为主外关系.例如.人和身份证的关系.每个人都对应一个身份证号.我们应该两个表.一个是关于人信息的表(Pers ...

  7. 基于 od 窗口的anti

    虽然 odadvance 这类的插件 , 使用驱动将 od 的窗口 进行 隐藏,使用enumwindow ,无法枚举到od的窗口, 但是依然可以 使用r3 的方法 , 对od 窗口检测 之后可以使用 ...

  8. Java Web 减少网络 IO、静态资源磁盘 IO 有效的办法--响应使用 GZIP( 压缩http请求与响应gzip压缩)

    (转载http://blog.csdn.net/hylclxy/article/details/7779662) 出于节约流量考虑, 客户端在向服务端发送request的时候对post数据进行gzip ...

  9. css table 布局 与 JavaScript 指定区域打印功能

      <!DOCTYPE html> <html lang="en"><head> <meta http-equiv="conte ...

  10. MySQL 百万级分页优化(Mysql千万级快速分页)

    以下分享一点我的经验 一般刚开始学SQL的时候,会这样写 : SELECT * FROM table ORDER BY id LIMIT 1000, 10; 但在数据达到百万级的时候,这样写会慢死 : ...