flume在抽取MySQL数据到kafka时报错,如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity = 
注意:Flume配置每行后边不能加注释,否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

  1. Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数

    package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...

  2. 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路

    最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...

  3. Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)

        前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...

  4. Apache Flume 学习笔记

    # 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...

  5. Apache Flume 1.7.0 发布,日志服务器

    Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...

  6. org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC

    2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...

  7. Apache Flume日志收集系统简介

    Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...

  8. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  9. Flafka: Apache Flume Meets Apache Kafka for Event Processing

    The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

  1. IOS7 UI Transition Guide 状态栏 statusbar

    本文转载至 http://blog.csdn.net/linzhiji/article/details/12233387 Redesign Your App for iOS 7 之 页面布局 iOS7 ...

  2. 爬虫实战【5】送福利!Python获取妹子图上的内容

    [插入图片,妹子图首页] 哈,只敢放到这个地步了. 今天给直男们送点福利,通过今天的代码,可以把你的硬盘装的满满的~ 下面就开始咯! 第一步:如何获取一张图片 假如我们知道某张图片的url,如何获取到 ...

  3. iOS 多线程之 NSOperation 的基本使用

    1.NSOperation,NSOperationQueue 简介 NSOperation,NSOperationQueue是苹果提供给我们的一套多线程解决方案.实际上 NSOperation.NSO ...

  4. 【转】NAS、DAS和SAN存储的特点及主要区别

    接触过存储设备的朋友应该对NAS.DAS和SAN存储不陌生,作为目前三种常见的存储方式,被广泛应用于企业存储设备中,那么NAS.DAS和SAN这三种存储分别有什么特点,又有何区别呢? 一.NAS存储特 ...

  5. 常用算法及其Python实现

    冒泡排序 ​ def bubble_sort(li):    for i in range(len(li)-1): # i表示第几趟        for j in range(len(li)-i-1 ...

  6. windows下redis的安装和启动

    Rides: //cmd管理员进入 // 运行 : redis-cli.exe //报错  :Redis (error) NOAUTH Authentication required.解决方法 // ...

  7. c# HttpClient获取网页源码

    #region 获取网页源码 public static string HttpClientGetHtmls(string url) { try { var client = new HttpClie ...

  8. 深度学习:Keras入门(一)之基础篇(转)

    转自http://www.cnblogs.com/lc1217/p/7132364.html 1.关于Keras 1)简介 Keras是由纯python编写的基于theano/tensorflow的深 ...

  9. C语言预处理命令的使用

    cppreference.com -> 预处理命令 -> 详细说明 预处理命令 #,## # 和 ## 操作符是和#define宏使用的. 使用# 使在#后的首个参数返回为一个带引号的字符 ...

  10. selenium之坑:点击后页面刷新重新获取刷新前的页面(StaleElementReferenceException:Message:Element not found in the cache...)

    循环点击一列链接,但只能点到第一个,第二个失败,这是为什么,因为第二个已经是新页面,当然找不到之前页面的元素,虽然元素没变,甚至是后退回来,页面都没有变,为什么是新页面,页面长的一样不代表是同一张页面 ...