org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
flume在抽取MySQL数据到kafka时报错,如下
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可
a1.channels.ch-.capacity =
注意:Flume配置每行后边不能加注释,否则会导致配置不生效
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章
- Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数
package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...
- 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路
最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...
- Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)
前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...
- Apache Flume 学习笔记
# 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...
- Apache Flume 1.7.0 发布,日志服务器
Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...
- org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC
2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...
- Apache Flume日志收集系统简介
Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...
- Apache Flume 1.7.0 各个模块简介
Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...
- Flafka: Apache Flume Meets Apache Kafka for Event Processing
The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...
随机推荐
- IOS7 UI Transition Guide 状态栏 statusbar
本文转载至 http://blog.csdn.net/linzhiji/article/details/12233387 Redesign Your App for iOS 7 之 页面布局 iOS7 ...
- 爬虫实战【5】送福利!Python获取妹子图上的内容
[插入图片,妹子图首页] 哈,只敢放到这个地步了. 今天给直男们送点福利,通过今天的代码,可以把你的硬盘装的满满的~ 下面就开始咯! 第一步:如何获取一张图片 假如我们知道某张图片的url,如何获取到 ...
- iOS 多线程之 NSOperation 的基本使用
1.NSOperation,NSOperationQueue 简介 NSOperation,NSOperationQueue是苹果提供给我们的一套多线程解决方案.实际上 NSOperation.NSO ...
- 【转】NAS、DAS和SAN存储的特点及主要区别
接触过存储设备的朋友应该对NAS.DAS和SAN存储不陌生,作为目前三种常见的存储方式,被广泛应用于企业存储设备中,那么NAS.DAS和SAN这三种存储分别有什么特点,又有何区别呢? 一.NAS存储特 ...
- 常用算法及其Python实现
冒泡排序 def bubble_sort(li): for i in range(len(li)-1): # i表示第几趟 for j in range(len(li)-i-1 ...
- windows下redis的安装和启动
Rides: //cmd管理员进入 // 运行 : redis-cli.exe //报错 :Redis (error) NOAUTH Authentication required.解决方法 // ...
- c# HttpClient获取网页源码
#region 获取网页源码 public static string HttpClientGetHtmls(string url) { try { var client = new HttpClie ...
- 深度学习:Keras入门(一)之基础篇(转)
转自http://www.cnblogs.com/lc1217/p/7132364.html 1.关于Keras 1)简介 Keras是由纯python编写的基于theano/tensorflow的深 ...
- C语言预处理命令的使用
cppreference.com -> 预处理命令 -> 详细说明 预处理命令 #,## # 和 ## 操作符是和#define宏使用的. 使用# 使在#后的首个参数返回为一个带引号的字符 ...
- selenium之坑:点击后页面刷新重新获取刷新前的页面(StaleElementReferenceException:Message:Element not found in the cache...)
循环点击一列链接,但只能点到第一个,第二个失败,这是为什么,因为第二个已经是新页面,当然找不到之前页面的元素,虽然元素没变,甚至是后退回来,页面都没有变,为什么是新页面,页面长的一样不代表是同一张页面 ...