前期博客

Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)

问题详情

  启动agent服务

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_MySearchAndReplaceInterceptor/  --conf-file conf_MySearchAndReplaceInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

  我这里,出现了这个错误

2017-07-29 10:17:51,006 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: fileSource started
2017-07-29 10:17:52,792 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:57)] Serializer = TEXT, UseRawLocalFileSystem = false
2017-07-29 10:17:55,094 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:231)] Creating hdfs://master:9000/data/types/20170729//run.1501294672792.data.tmp
2017-07-29 10:17:55,842 (hdfs-hdfsSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-07-29 10:18:00,495 (pool-5-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:352)] Failed while running command: tail -F /usr/local/log/server.log
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:127)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:341)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-07-29 10:18:00,544 (timedFlushExecService21-0) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:327)] Exception occured when processing event batch
org.apache.flume.ChannelException: java.lang.InterruptedException
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:381)
at org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:254)
at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:323)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

  解决办法

  

agent1.channels.memoryChannel.keep-alive = 60
agent1.channels.memoryChannel.capacity = 1000000

  然后,再来修改

修改java最大内存大小
vi bin/flume-ng
JAVA_OPTS="-Xmx1024m"

  改为

  即,修改后之后,再次运行

[hadoop@master flume-1.7.0]$ bin/flume-ng agent --conf conf_MySearchAndReplaceInterceptor/  --conf-file conf_MySearchAndReplaceInterceptor/flume-conf.properties --name agent1  -Dflume.root.logger=INFO,console

  上述错误,得以解决了。

  参考博客

flume-ng 问题处理(1)

Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)的更多相关文章

  1. Flume报 Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight

    报这个错误 需要一个是flume堆内存不够.还有一个就是把channel的容器调大 在channel加配置 type - 组件类型名称必须是memory capacity 100 存储在 Channe ...

  2. Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)

    不多说,直接上干货! 一.自定义拦截器类型必须是:类全名$内部类名,其实就是内部类名称 如:zhouls.bigdata.MySearchAndReplaceInterceptor$Builder 二 ...

  3. 使用eclipse启动系统时报错“ java.lang.OutOfMemoryError: PermGen space”问题的解决

    转载请注明出处:http://blog.csdn.net/dongdong9223/article/details/76571611 本文出自[我是干勾鱼的博客] 有的时候,使用eclipse启动系统 ...

  4. Elasticsearch集群状态健康值处于red状态问题分析与解决(图文详解)

      问题详情 我的es集群,开启后,都好久了,一直报red状态??? 问题分析 有两个分片数据好像丢了.   不知道你这数据怎么丢的. 确认下本地到底还有没有,本地要是确认没了,那数据就丢了,删除索引 ...

  5. PHP生成页面二维码解决办法?详解

    随着科技的进步,二维码应用领域越来越广泛,今天我给大家分享下如何使用PHP生成二维码,以及如何生成中间带LOGO图像的二维码. 具体工具: phpqrcode.php内库:这个文件可以到网上下载,如果 ...

  6. Flume启动报错[ERROR - org.apache.flume.sink.hdfs. Hit max consecutive under-replication rotations (30); will not continue rolling files under this path due to under-replication解决办法(图文详解)

    前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)   问题详情 -- ::, (SinkRunner-PollingRunner-Default ...

  7. Flume启动时报错Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解决办法(图文详解)

    前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 -- ::, (agent-shutdown-hook) [INFO - org.a ...

  8. cloudera manager的7180 web界面访问不了的解决办法(图文详解)

    说在前面的话 我的机器是总共4台,分别为ubuntucmbigdata1.ubuntucmbigdata2.ubuntucmbigdata3和ubuntucmbigdata4.(注意啦,以下是针对Ub ...

  9. Windows环境下执行hadoop命令出现Error: JAVA_HOME is incorrectly set Please update D:\SoftWare\hadoop-2.6.0\conf\hadoop-env.cmd错误的解决办法(图文详解)

    不多说,直接上干货! 导读   win下安装hadoop 大家,别小看win下的安装大数据组件和使用  玩过dubbo和disconf的朋友们,都知道,在win下安装zookeeper是经常的事   ...

随机推荐

  1. Linux buffer and cache

    A buffer is something that has yet to be "written" to disk. A cache is something that has ...

  2. easyui页签更新

    1.首先引入这个js文件 <script src="/Scripts/tabs.js" type="text/javascript"></sc ...

  3. Storm的StreamID使用样例(版本1.0.2)

    随手尝试了一下StreamID的的用法.留个笔记. ==数据样例== { "Address": "小桥镇小桥中学对面", "CityCode" ...

  4. PropertiesConfiguration 修改配置文件的信息,不打乱顺序

    需引入jar包 <!-- https://mvnrepository.com/artifact/commons-configuration/commons-configuration --> ...

  5. 为啥final类型的map或者arraylist可以修改数据 而final类型的String变量不可以修改数据呢

    比如 final   Map  map =new  HashMap();    可以往map里put数据final   List  list =new  ArrayList();   可以往list里 ...

  6. (3)WePHP控制器与模型

    MVC的先进理念就是程序流程 数据库操作 模板制作三个流程分离 大M函数会把模型引入到当前控制器的方法中,注意这个"excel"对应M文件夹的excel.php 同时也是整个exc ...

  7. javascript总结39:DOM 中常用的表单元素的属性

    1 常用操作元素: value 用于大部分表单元素的内容获取(option除外) type 可以获取input标签的类型(输入框或复选框等) disabled 禁用属性 checked 复选框选中属性 ...

  8. C# JackLib系列之自定义鼠标风格的实现

    在我们开发的过程中,有时需要我们来自定义鼠标的形状和大小,刚巧前一阵子正好用到了这个技术,找了好多资料,基本上都是黑白色的鼠标风格实现,而我要的则是自定义大小和彩色风格的光标样式.百度上的资源又太少, ...

  9. 编写高质量代码改善C#程序的157个建议——建议153:若抛出异常,则必须要注释

    建议153:若抛出异常,则必须要注释 有一种必须加注释的场景,即使异常.如果API抛出异常,则必须给出注释.调用者必须通过注释才能知道如何处理那些专有的异常.通常,即便良好的命名也不可能告诉我们方法会 ...

  10. winsock的io模型(终极篇)

    最近在看服务器框架的搭建,看了不少,都是零零碎碎的,觉得看的差不多了,可以写点最后的总结了,然后,竟然发现了这篇文章,总结做的特别好,肯定比我总结写要好多了,所以我也就不写了,直接转吧...... 套 ...