org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
flume在抽取MySQL数据到kafka时报错,如下
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可
a1.channels.ch-.capacity =
注意:Flume配置每行后边不能加注释,否则会导致配置不生效
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章
- Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数
package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...
- 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路
最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...
- Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...
- Apache Flume 学习笔记
# 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...
- Apache Flume 1.7.0 发布,日志服务器
Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...
- org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC
2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...
- Apache Flume日志收集系统简介
Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...
- Apache Flume 1.7.0 各个模块简介
Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...
- Flafka: Apache Flume Meets Apache Kafka for Event Processing
The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...
随机推荐
- E - I Hate It(基础线段树)
E - I Hate It Time Limit:3000MS Memory Limit:32768KB 64bit IO Format:%I64d & %I64u Descr ...
- ECMAScript6箭头函数ArrowFunction"=>"
一.说明 ECMAScript6可以用箭头"=>"定义函数.x => x * x或(x) => {return x * x;}与匿名函数function(x){r ...
- SQLServer与ASP中DATEDIFF函数区别
一.SQLServer: 格式: DATEDIFF(interval,startdate,enddate) interval参数: 年-yy,yyyy 季度-qq,q 月-mm,m 年中的日-dy,y ...
- 避免每次都用sudo使用docker
默认安装完 docker 后,每次执行 docker 都需要运行 sudo 命令,非常浪费时间影响效率.如果不跟 sudo,直接执行 docker images 命令会有如下问题: FATA[0000 ...
- mac 识别压缩文件类型
file -z b.zip
- Hadoop createSnapshot和deleteSnapshot命令
概述 HDFS快照是文件系统的只读时间点副本. 可以对文件系统的子树或整个文件系统进行快照. 快照的一些常见用例是数据备份,防止用户错误和灾难恢复.HDFS快照的实现是高效的: 快照创建是即时的:成本 ...
- (4.12)全面解析-SQL事务+隔离级别+阻塞+死锁
30分钟全面解析-SQL事务+隔离级别+阻塞+死锁 转自:https://blog.csdn.net/slowlifes/article/details/52752735 2016年10月07日 23 ...
- SMW0 上传问题?
*SMW0 和 OAOR 的区别在哪3个方面? SMW0 上传 出现: 没有指派至MIME 类型
- virt-viewer的简单使用
virt-viewer 简介: virt-viewer是一个用于显示虚拟机的图形控制台的最小工具. 控制台使用VNC或SPICE访问协议. 可以基于其名称,ID或UUID来引用guest虚拟机.如果 ...
- JAVA 遍历文件夹下文件并更改文件名称
周末因为一些原因,需要批量更改一些文件的名称,使其随机,就随手写了点代码. 增加一个随机字母: public static void changeName(String path){ File fil ...