1. 错误提示

  

-- ::, (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)] Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=channel1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:)
at org.apache.flume.channel.file.Log.replay(Log.java:)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
... more
-- ::, (pool--thread-) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:)] FATAL: Spool Directory source source1: { spoolDir: /home/hadoop_admin/movielog }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.lang.IllegalStateException: Channel closed [channel=channel1]. Due to java.io.EOFException: null
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:)
at org.apache.flume.channel.file.Log.replay(Log.java:)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:)
... more

 配置文件:

  

agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1 # Each channel's type is defined.
agent1.channels.channel1.type = file
agent1.channels.channel1.checkpointDir = /home/hadoop_admin/flumeTemp/fchannel/spool/checkpoint
agent1.channels.channel1.dataDirs = /home/hadoop_admin/flumeTemp/fchannel/spool/data
agent1.channels.channel1.capacity =
agent1.channels.channel1.keep-alive =
agent1.channels.channel1.write-timeout =
agent1.channels.channel1.checkpoint-timeout = # For each one of the sources, the type is defined
agent1.sources.source1.type = spooldir
agent1.sources.source1.inputCharset = GBK
agent1.sources.source1.spoolDir =/home/hadoop_admin/movielog
agent1.sources.source1.fileHeader = true
agent1.sources.source1.deletePolicy = immediate
agent1.sources.source1.batchSize =
agent1.sources.source1.channels = channel1 # Each sink's type must be defined
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.hdfs.path = hdfs://master:9000/flumeTest
agent1.sinks.sink1.hdfs.filePrefix = master-
agent1.sinks.sink1.hdfs.writeFormat = Text
agent1.sinks.sink1.hdfs.fileType = DataStream
agent1.sinks.sink1.hdfs.rollInterval =
agent1.sinks.sink1.hdfs.idleTimeout =
agent1.sinks.sink1.hdfs.rollCount =
agent1.sinks.sink1.hdfs.rollSize =
agent1.sinks.sink1.hdfs.batchSize =
agent1.sinks.sink1.hdfs.callTimeout =

2. 解决方法

  链接: http://serverfault.com/questions/690588/flume-error-log-while-using-filechannel

  只要把配置文件中的关于channel配置checkpointDir和dataDir目录清空就可以

flume file channel 异常解决的更多相关文章

  1. [bigdata] flume file channel CPU消耗比 memory channel高的原因

    https://www.quora.com/Why-does-flume-take-more-resource-CPU-when-file-channel-is-used-compared-to-wh ...

  2. flume 报File Channel transaction capacity cannot be greater than the capacity of the channel capacity错误

    今天在部署flume集群时,在启动collector服务器没报错,启动agent服务器报错: File Channel transaction capacity cannot be greater t ...

  3. 安装Flume的时候出现File Channel transaction capacity cannot be greater than the capacity of the channel capacity -解决方案 摘自网络

    部署flume集群时,在启动collector服务器没报错,启动agent服务器报错: File Channel transaction capacity cannot be greater than ...

  4. 实时事件统计项目:优化flume:用file channel代替mem channel

    背景:利用kafka+flume+morphline+solr做实时统计. solr从12月23号开始一直没有数据.查看日志发现,因为有一个同事加了一条格式错误的埋点数据,导致大量error. 据推断 ...

  5. [异常解决] How to build a gcc toolchain for nRF51 on linux (very detailed!!!)

    1.Install gcc-arm-none-eabi https://devzone.nordicsemi.com/tutorials/7/This link shows that developm ...

  6. STL 跨模块 调用 异常 解决

    本文为转载别人的,以作收藏之用 百度了一天,现在把结论放上边: 1.不要用STL(std::string属于STL)来跨模块传输数据,例如:dll(so)之间,dll(so)和exe(elf)之间. ...

  7. Android 异常解决方法【汇总】

    (1)异常:Android中引入第三方Jar包的方法(Java.lang.NoClassDefFoundError解决办法) 1.在工程下新建lib文件夹,将需要的第三方包拷贝进来.2.将引用的第三方 ...

  8. java启动RabbitMQ消息报异常解决办法

    启动SpringCloud微服务,RabbitMQ报如下异常: 2019-08-12 18:15:49.543 ERROR 53096 --- [68.252.131:5672] o.s.a.r.c. ...

  9. MySQL提示:The server quit without updating PID file问题的解决办法(转载)

    MySQL提示:The server quit without updating PID file问题的解决办法 今天网站web页面提交内容到数据库,发现出错了,一直提交不了,数找了下原因,发现数据写 ...

随机推荐

  1. IOS- 自定义 UIButton

    #pragma mark init方法内部默认会调用initWithFrame: - (id)initWithFrame:(CGRect)frame { self = [super initWithF ...

  2. php策略模式的研究

    <?php abstract class Moshi{     private $num;     public $price;     const Ted=1;     const Sed=2 ...

  3. 详解web.xml中元素的加载顺序

    一.背景 最近在项目中遇到了启动时出现加载service注解注入失败的问题,后来经过不懈努力发现了是因为web.xml配置文件中的元素加载顺序导致的,那么就抽空研究了以下tomcat在启动时web.x ...

  4. Redis自定义动态字符串(sds)模块(一)

    Redis开发者在开发过程中没有使用系统的原始字符串,而是使用了自定义的sds字符串,这个模块的编写是在文件:sds.h和sds.c文件中.Redis自定义的这个字符串好像也不是很复杂,远不像ngin ...

  5. 关于ActionContext.getContext()的用法心得

    转: 为了避免与Servlet API耦合在一起,方便Action类做单元测试,Struts 2对HttpServletRequest.HttpSession和ServletContext进行了封装, ...

  6. Powershell实例小结(服务管理)

    有关服务管理的具体实例脚本如下: #$lists="1.1.1.1","2.2.2.2" #远程ip列表 foreach ($list in $lists){ ...

  7. Java -- 找不到或无法加载主类

    原文:http://wenku.baidu.com/link?url=5nS1GEaePn-hmtAg6xXdJvtt9Z89JQsakhqSv8fambaJY2t9nKPtf3hXFpjW-BtD9 ...

  8. python 取两数的百分比

    Python (r266:, Jan , ::) [GCC (Red Hat -)] on linux2 Type "help", "copyright", & ...

  9. ReentrantLock和synchronized两种锁定机制

    ReentrantLock和synchronized两种锁定机制 >>应用synchronized同步锁 把代码块声明为 synchronized,使得该代码具有 原子性(atomicit ...

  10. SQL链表查询 数据库为空

    查询出数据为空,解决方案:链表 对应字段长度不一致.