org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count

flume在抽取MySQL数据到kafka时报错，如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: Failed to publish events

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    ...  more

MemoryChannel的事务的Put队列满

解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity =

注意：Flume配置每行后边不能加注释，否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数
package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...
一个Flume 异常（Put queue for MemoryTransaction of capacity 100 full）的排查和解决思路
最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...
Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法（图文详解）
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情启动agent服务 [hadoop@master flume-1.7.0]$ ...
Apache Flume 学习笔记
# 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...
Apache Flume 1.7.0 发布，日志服务器
Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...
org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC
2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...
Apache Flume日志收集系统简介
Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...
Apache Flume 1.7.0 各个模块简介
Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...
Flafka: Apache Flume Meets Apache Kafka for Event Processing
The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

PHP 可以获取客户端哪些访问信息
php是一种弱类型的程序语言,但是最web的在程序语言中有系统全局函数: $_SERVER <?php echo "".$_SERVER['PHP_SELF'];#当前正在 ...
node.js中的路由（url）初步
1.建立n4_root.js var http = require('http'); var url = require('url'); //这是node.js中自带的var router = req ...
bsd socket 网络通讯必备工具类
传输数据的时候都要带上包头,包头有简单的又复杂的,简单的只要能指明数据的长度就够了. 这里我写了一个工具类,可以方便地将整型的数据长度转换为长度为 4 的字节数组. 另一方面,可以方便的将长度为 4 ...
【BZOJ1925】[Sdoi2010]地精部落组合数+DP
[BZOJ1925][Sdoi2010]地精部落 Description 传说很久以前,大地上居住着一种神秘的生物:地精. 地精喜欢住在连绵不绝的山脉中.具体地说,一座长度为 N 的山脉 H可分为从 ...
cocos2d-x中关于touch事件的响应
原作者:有缘人来源:新浪微博地址:http://blog.sina.com.cn/s/blog_6ac2c7260102vvdu.html 一.touch事件响应分为单点触摸响应和多点触摸响应. ...
九度OJ 1359：大魏树遍历（树）
时间限制:1 秒内存限制:32 兆特殊判题:否提交:299 解决:29 题目描述: 大魏是JOBDU技术组里最喜欢折腾的一个了,单反.骑车.改九度页面,当然还有YY prado.我们姑且先把这些 ...
根据字段表自动创建表SQL
create PROC CreateTableSql ) AS ) ,) ,) ) ,) ,) ,@IsIdentity bit ,@IsNull bit ,@IsPrimaryKey bit ),) ...
Docker容器/镜像查看及删除操作
列出所有正在运行的容器 docker ps 暂停容器 docker stop <name> 删除容器 docker rm <name> 停止所有container docker ...
random模块（随机数库）
random random.random random.random()用于生成一个0到1的随机浮点数: 0 <= n < 1.0 random.uniform random.unifor ...
Java Web项目在Mac系统上启动时提示nodename nor servname provided
今天一不小心更新了Mac系统,然后在启动Java Web项目的时候,提示了java.net.UnknownHostException: MAC-mini-local nodename nor serv ...

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

随机推荐

热门专题