Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)
前期博客
Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)
问题详情
启动agent服务
[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_MySearchAndReplaceInterceptor/ --conf-file conf_MySearchAndReplaceInterceptor/flume-conf.properties --name agent1 -Dflume.root.logger=INFO,console
我这里,出现了这个错误

2017-07-29 10:17:51,006 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: fileSource started
2017-07-29 10:17:52,792 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)] Serializer = TEXT, UseRawLocalFileSystem = false
2017-07-29 10:17:55,094 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170729//run.1501294672792.data.tmp
2017-07-29 10:17:55,842 (hdfs-hdfsSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-07-29 10:18:00,495 (pool-5-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:352)] Failed while running command: tail -F /usr/local/log/server.log
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:127)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:341)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-07-29 10:18:00,544 (timedFlushExecService21-0) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:327)] Exception occured when processing event batch
org.apache.flume.ChannelException: java.lang.InterruptedException
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
at org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:254)
at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:323)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

解决办法
agent1.channels.memoryChannel.keep-alive = 60
agent1.channels.memoryChannel.capacity = 1000000
然后,再来修改
修改java最大内存大小
vi bin/flume-ng
JAVA_OPTS="-Xmx1024m"
改为
即,修改后之后,再次运行
[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_MySearchAndReplaceInterceptor/ --conf-file conf_MySearchAndReplaceInterceptor/flume-conf.properties --name agent1 -Dflume.root.logger=INFO,console
上述错误,得以解决了。
参考博客
Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)的更多相关文章
- Flume报 Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
报这个错误 需要一个是flume堆内存不够.还有一个就是把channel的容器调大 在channel加配置 type - 组件类型名称必须是memory capacity 100 存储在 Channe ...
- Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)
不多说,直接上干货! 一.自定义拦截器类型必须是:类全名$内部类名,其实就是内部类名称 如:zhouls.bigdata.MySearchAndReplaceInterceptor$Builder 二 ...
- 使用eclipse启动系统时报错“ java.lang.OutOfMemoryError: PermGen space”问题的解决
转载请注明出处:http://blog.csdn.net/dongdong9223/article/details/76571611 本文出自[我是干勾鱼的博客] 有的时候,使用eclipse启动系统 ...
- Elasticsearch集群状态健康值处于red状态问题分析与解决(图文详解)
问题详情 我的es集群,开启后,都好久了,一直报red状态??? 问题分析 有两个分片数据好像丢了. 不知道你这数据怎么丢的. 确认下本地到底还有没有,本地要是确认没了,那数据就丢了,删除索引 ...
- PHP生成页面二维码解决办法?详解
随着科技的进步,二维码应用领域越来越广泛,今天我给大家分享下如何使用PHP生成二维码,以及如何生成中间带LOGO图像的二维码. 具体工具: phpqrcode.php内库:这个文件可以到网上下载,如果 ...
- Flume启动报错[ERROR - org.apache.flume.sink.hdfs. Hit max consecutive under-replication rotations (30); will not continue rolling files under this path due to under-replication解决办法(图文详解)
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 -- ::, (SinkRunner-PollingRunner-Default ...
- Flume启动时报错Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解决办法(图文详解)
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 -- ::, (agent-shutdown-hook) [INFO - org.a ...
- cloudera manager的7180 web界面访问不了的解决办法(图文详解)
说在前面的话 我的机器是总共4台,分别为ubuntucmbigdata1.ubuntucmbigdata2.ubuntucmbigdata3和ubuntucmbigdata4.(注意啦,以下是针对Ub ...
- Windows环境下执行hadoop命令出现Error: JAVA_HOME is incorrectly set Please update D:\SoftWare\hadoop-2.6.0\conf\hadoop-env.cmd错误的解决办法(图文详解)
不多说,直接上干货! 导读 win下安装hadoop 大家,别小看win下的安装大数据组件和使用 玩过dubbo和disconf的朋友们,都知道,在win下安装zookeeper是经常的事 ...
随机推荐
- Linux buffer and cache
A buffer is something that has yet to be "written" to disk. A cache is something that has ...
- easyui页签更新
1.首先引入这个js文件 <script src="/Scripts/tabs.js" type="text/javascript"></sc ...
- Storm的StreamID使用样例(版本1.0.2)
随手尝试了一下StreamID的的用法.留个笔记. ==数据样例== { "Address": "小桥镇小桥中学对面", "CityCode" ...
- PropertiesConfiguration 修改配置文件的信息,不打乱顺序
需引入jar包 <!-- https://mvnrepository.com/artifact/commons-configuration/commons-configuration --> ...
- 为啥final类型的map或者arraylist可以修改数据 而final类型的String变量不可以修改数据呢
比如 final Map map =new HashMap(); 可以往map里put数据final List list =new ArrayList(); 可以往list里 ...
- (3)WePHP控制器与模型
MVC的先进理念就是程序流程 数据库操作 模板制作三个流程分离 大M函数会把模型引入到当前控制器的方法中,注意这个"excel"对应M文件夹的excel.php 同时也是整个exc ...
- javascript总结39:DOM 中常用的表单元素的属性
1 常用操作元素: value 用于大部分表单元素的内容获取(option除外) type 可以获取input标签的类型(输入框或复选框等) disabled 禁用属性 checked 复选框选中属性 ...
- C# JackLib系列之自定义鼠标风格的实现
在我们开发的过程中,有时需要我们来自定义鼠标的形状和大小,刚巧前一阵子正好用到了这个技术,找了好多资料,基本上都是黑白色的鼠标风格实现,而我要的则是自定义大小和彩色风格的光标样式.百度上的资源又太少, ...
- 编写高质量代码改善C#程序的157个建议——建议153:若抛出异常,则必须要注释
建议153:若抛出异常,则必须要注释 有一种必须加注释的场景,即使异常.如果API抛出异常,则必须给出注释.调用者必须通过注释才能知道如何处理那些专有的异常.通常,即便良好的命名也不可能告诉我们方法会 ...
- winsock的io模型(终极篇)
最近在看服务器框架的搭建,看了不少,都是零零碎碎的,觉得看的差不多了,可以写点最后的总结了,然后,竟然发现了这篇文章,总结做的特别好,肯定比我总结写要好多了,所以我也就不写了,直接转吧...... 套 ...