Flume wasn't able to parse timestamp header
来自:http://caiguangguang.blog.51cto.com/1652935/1384187
flume bucketpath的bug一例
测试的配置文件:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
agent-server1.sources= testtail agent-server1.sinks = hdfs-sink agent-server1.channels= hdfs-channel agent-server1.sources.testtail. type = netcat agent-server1.sources.testtail.bind = localhost agent-server1.sources.testtail.port = 9999 agent-server1.sinks.hdfs-sink.hdfs.kerberosPrincipal = hdfs /_HOST @KERBEROS_HADOOP agent-server1.sinks.hdfs-sink.hdfs.kerberosKeytab = /home/vipshop/conf/hdfs .keytab agent-server1.channels.hdfs-channel. type = memory agent-server1.channels.hdfs-channel.capacity = 200000000 agent-server1.channels.hdfs-channel.transactionCapacity = 10000 agent-server1.sinks.hdfs-sink. type = hdfs agent-server1.sinks.hdfs-sink.hdfs.path = hdfs: //bipcluster/tmp/flume/ %Y%m%d agent-server1.sinks.hdfs-sink.hdfs.rollInterval = 60 agent-server1.sinks.hdfs-sink.hdfs.rollSize = 0 agent-server1.sinks.hdfs-sink.hdfs.rollCount = 0 agent-server1.sinks.hdfs-sink.hdfs.threadsPoolSize = 10 agent-server1.sinks.hdfs-sink.hdfs.round = false agent-server1.sinks.hdfs-sink.hdfs.roundValue = 30 agent-server1.sinks.hdfs-sink.hdfs.roundUnit = minute agent-server1.sinks.hdfs-sink.hdfs.batchSize = 100 agent-server1.sinks.hdfs-sink.hdfs.fileType = DataStream agent-server1.sinks.hdfs-sink.hdfs.writeFormat = Text agent-server1.sinks.hdfs-sink.hdfs.callTimeout = 60000 agent-server1.sinks.hdfs-sink.hdfs.idleTimeout = 100 agent-server1.sinks.hdfs-sink.hdfs.filePrefix = ip agent-server1.sinks.hdfs-sink.channel = hdfs-channel agent-server1.sources.testtail.channels = hdfs-channel |
在启动服务后,使用telnet进行测试,发现如下报错:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
14 / 03 / 24 18 : 03 : 07 ERROR hdfs.HDFSEventSink: process failed java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header ( for example using TimestampInterceptor source interceptor). at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java: 160 ) at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java: 343 ) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java: 392 ) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java: 68 ) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java: 147 ) at java.lang.Thread.run(Thread.java: 662 ) Caused by: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java: 375 ) at java.lang.Long.valueOf(Long.java: 525 ) at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java: 158 ) ... 5 more 14 / 03 / 24 18 : 03 : 07 ERROR flume.SinkRunner: Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header ( for example using TimestampInterceptor source interceptor). at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java: 461 ) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java: 68 ) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java: 147 ) at java.lang.Thread.run(Thread.java: 662 ) Caused by: java.lang.RuntimeException: Flume wasn 't able to parse timestamp header in the event to resolve time based bucketing. Please check that you' re correctly populating timestamp header ( for example using TimestampInterceptor source interceptor). at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java: 160 ) at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java: 343 ) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java: 392 ) ... 3 more Caused by: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java: 375 ) at java.lang.Long.valueOf(Long.java: 525 ) at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java: 158 ) ... 5 more |
从调用栈的信息来看,错误出在org.apache.flume.formatter.output.BucketPath类的replaceShorthand方法。
在org.apache.flume.sink.hdfs.HDFSEventSink类中,使用process方法来生成hdfs的url,其中主要是调用了BucketPath类的escapeString方法来进行字符的转换,并最终调用了replaceShorthand方法。
其中replaceShorthand方法的相关代码如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
public static String replaceShorthand( char c, Map<String, String> headers, TimeZone timeZone, boolean needRounding, int unit, int roundDown) { String timestampHeader = headers.get( "timestamp" ); long ts; try { ts = Long.valueOf(timestampHeader); } catch (NumberFormatException e) { throw new RuntimeException( "Flume wasn't able to parse timestamp header" + " in the event to resolve time based bucketing. Please check that" + " you're correctly populating timestamp header (for example using" + " TimestampInterceptor source interceptor)." , e); } if (needRounding){ ts = roundDown(roundDown, unit, ts); } ........ |
从代码中可以看到,timestampHeader 的值如果取不到,在向ts赋值时就会报错。。
这其实是flume的一个bug,bug id:
https://issues.apache.org/jira/browse/FLUME-1419
解决方法有3个:
1.更改配置,更新hdfs文件的路径格式
1
|
agent-server1.sinks.hdfs-sink.hdfs.path = hdfs: //bipcluster/tmp/flume |
但是这样就不能按天来存放日志了
2.通过更改相关的代码
(patch:https://issues.apache.org/jira/secure/attachment/12538891/FLUME-1419.patch)
如果在headers中获取不到timestamp的值,就给它一个当前timestamp的值。
相关代码:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
String timestampHeader = headers.get( "timestamp" ); long ts; try { if (timestampHeader == null) { ts = System.currentTimeMillis(); } else { ts = Long.valueOf(timestampHeader); } } catch (NumberFormatException e) { throw new RuntimeException( "Flume wasn't able to parse timestamp header" + " in the event to resolve time based bucketing. Please check that" + " you're correctly populating timestamp header (for example using" + " TimestampInterceptor source interceptor)." , e); } |
3.为source定义基于timestamp的interceptors
在配置中增加两行即可:
1
2
|
agent-server1.sources.testtail.interceptors = i1 agent-server1.sources.testtail.interceptors.i1. type = org.apache.flume.interceptor.TimestampInterceptor$Builder |
一个技巧:
在debug flume的问题时,可以在flume的启动参数中设置把debug日志打到console中。
1
|
-Dflume.root.logger=DEBUG,console,LOGFILE |
Flume wasn't able to parse timestamp header的更多相关文章
- 关于flume中涉及到时间戳的错误解决,Expected timestamp in the Flume even
在搭建flume集群收集日志写入hdfs时发生了下面的错误: java.lang.NullPointerException: Expected timestamp in the Flume event ...
- 当我new class的时候,提示以下错误: Unable to parse template "Class" Error message: This template did not produce a Java class or an interface Error parsing file template: Unable to find resource 'Package Header.j
你肯定修改过class的template模板,改回去就好了 #if (${PACKAGE_NAME} && ${PACKAGE_NAME} != "")packag ...
- Flume架构
Flume是Cloudera提供的一个高可用的,高可靠的,分布式的海量日志采集.聚合和传输的系统: Flume 介绍 Flume是由cloudera软件公司产出的高可用.高可靠.分布式的海量日志收集系 ...
- netty 网关 flume 提交数据 去除透明 批处理 批提交 cat head tail 结合 管道显示行号
D:\javaNettyAction\NettyA\src\main\java\com\test\HexDumpProxy.java package com.test; import io.netty ...
- TimeStamp
private void Form1_Load(object sender, EventArgs e) { textBox1.Text= GenerateTimeStamp(System.DateTi ...
- Flume官方文档翻译——Flume 1.7.0 User Guide (unreleased version)(一)
Flume 1.7.0 User Guide Introduction(简介) Overview(综述) System Requirements(系统需求) Architecture(架构) Data ...
- c# datetime与 timeStamp(unix时间戳) 互相转换
/// <summary> /// Unix时间戳转为C#格式时间 /// </summary> /// <param name="timeStamp" ...
- webMagic解析淘宝cookie 提示Invalid cookie header
webMagic解析淘宝cookie 提示Invalid cookie header 在使用webMagic框架做爬虫爬取淘宝极又家页面时候一直提醒cookie设置不可用如下图 淘宝的验证特别严重,c ...
- c# datetime与 timeStamp时间戳 互相转换
将时间格式转化为一个int类型 // ::26时间转完后为:1389675686数字 为什么使用时间戳? 关于Unix时间戳,大概是这个意思,从1970年0时0分0秒开始到现在的秒数.使用它来获得的是 ...
随机推荐
- 0056 Spring MVC如何接收浏览器传递来的请求参数--request--形参--实体类封装
浏览器总会向服务器传递一些参数,那么Spring MVC如何接收这些参数? 先写个简单的html,向服务器传递一些书籍信息,如下: <!DOCTYPE html> <html> ...
- centos7搭建.netcore运行环境
开发环境介绍 1.操作系统:Windows Server 2008 R2 Enterprise 2.IDE:VisualStudio2017 3..Net Core 2.0 SDK 本文假设你已经满足 ...
- Window 下安装
Window 下安装 下载地址:https://github.com/MSOpenTech/redis/releases Redis 支持 32 位和 64 位.这个需要根据你系统平台的实际情况选择, ...
- VPS 虚拟私有主机 Virtual Private Server
VPS技术介绍 利用最新虚拟化技术Xen在一台物理服务器上创建多个相互隔离的虚拟私有主机(“Virtual Private Server”简称 “VPS”).这些VPS以最大化的效率共享硬件.软件许可 ...
- 分频器VHDL描述
在数字电路中,常需要对较高频率的时钟进行分频操作,得到较低频率的时钟信号.我们知道,在硬件电路设计中时钟信号时非常重要的. 下面我们介绍分频器的VHDL描述,在源代码中完成对时钟信号CLK的2分 ...
- php输出mysqli查询出来的结果
php连接mysql我有文章已经写过了,这篇文章主要是介绍从mysql中查询出结果之后怎么输出的问题. 一:mysqli_fetch_row(); 查询结果:array([0]=>小王) 查询: ...
- if....else的基本用法
if....else...是基本流程控制语句 1.基本格式: if(条件){ }else if(条件){ }else if(条件){ } ........ else{ } 解释:其中else if.e ...
- java 字符串,字符数组,list间的转化
1.关于java.lang.string.split xxx.split()方法可以将一个字符串分割为子字符串,然后将结果作为字符串数组返回. 2.字符串转字符数组 String str =" ...
- HostMonitor监控主机状态
HostMonitor 可以对windows和linux下的主机进行很多信息的监控,还提供web方式查看
- freetds简介、安装、配置及使用介绍
什么是FreeTDS 简单的说FreeTDS是一个程序库,可以实现在Linux系统下访问微软的SQL数据库! FreeTDS 是一个开源(如果你喜欢可以称为自由)的程序库,是TDS(表列数据流 )协议 ...