org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
flume在抽取MySQL数据到kafka时报错,如下
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可
a1.channels.ch-.capacity =
注意:Flume配置每行后边不能加注释,否则会导致配置不生效
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章
- Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数
package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...
- 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路
最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...
- Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...
- Apache Flume 学习笔记
# 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...
- Apache Flume 1.7.0 发布,日志服务器
Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...
- org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC
2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...
- Apache Flume日志收集系统简介
Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...
- Apache Flume 1.7.0 各个模块简介
Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...
- Flafka: Apache Flume Meets Apache Kafka for Event Processing
The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...
随机推荐
- windows下 兼容Python2和Python3
windows下同时安装了python2和python3时,都可以配置环境变量,如果在命令行里输入python命令,windows会去环境变量里寻找Python的安装位置,如果先找到pytoon2的, ...
- 【BZOJ2799】[Poi2012]Salaries 乱搞
[BZOJ2799][Poi2012]Salaries Description 给出一棵n个结点的有根树,结点用正整数1~n编号.每个结点有一个1~n的正整数权值,不同结点的权值不相同,并且一个结点的 ...
- sparse matrix
w https://en.wikipedia.org/wiki/Sparse_matrix 稀疏矩阵存储格式总结+存储效率对比:COO,CSR,DIA,ELL,HYB - Bin的专栏 - 博客园ht ...
- CalendarUtil 日期操作工具类
版权声明:本文为博主原创文章,未经博主允许不得转载. [java] view plain copy import java.util.Calendar; import java.text.DateFo ...
- CF85D Sum of Medians
CF85D Sum of Medians 题意翻译 一个集合,初始为空.现有三个操作: 1.add:向集合里加入数x,保证加入前集合中没有数x: 2.del:从集合中删除数x,保证删除前集合中有x: ...
- chrome浏览器自动填充失效问题
现在浏览器带有自动填充,一般在input标签中增加autocomplete="off" 可以进行控制,off代表不填充,on代表填充,这个属性也可以放在form标签中,对所有的in ...
- Eclipse解决JSP页面乱码问题
解决方案 1.首先选择 Window --> Preferences --> General--> Content Types --> text (点击左边的+,显示 ...
- 1.6 使用电脑测试MC20的读取带中文短信功能
需要准备的硬件 MC20开发板 1个 https://item.taobao.com/item.htm?id=562661881042 GSM/GPRS天线 1根 https://item.taoba ...
- Apache Shiro:【2】与SpringBoot集成完成登录验证
Apache Shiro:[2]与SpringBoot集成完成登录验证 官方Shiro文档:http://shiro.apache.org/documentation.html Shiro自定义Rea ...
- hadoop linux 杂记
切换到root su 修改sudo sudo + 命令 --> root权限 + 命令 su root vim /etc/sudoers ...