flume在抽取MySQL数据到kafka时报错，如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: Failed to publish events

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    ...  more


MemoryChannel的事务的Put队列满

解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity = 
注意：Flume配置每行后边不能加注释，否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

  1. Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数

    package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...

  2. 一个Flume 异常（Put queue for MemoryTransaction of capacity 100 full）的排查和解决思路

    最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...

  3. Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法（图文详解）

        前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...

  4. Apache Flume 学习笔记

    # 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...

  5. Apache Flume 1.7.0 发布，日志服务器

    Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...

  6. org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC

    2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...

  7. Apache Flume日志收集系统简介

    Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...

  8. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  9. Flafka: Apache Flume Meets Apache Kafka for Event Processing

    The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

  1. Notepad++ 64位 插件管理

    notepad++ 64bit 没有插件管理,如何添加呢? 1.访问https://github.com/bruderstein/nppPluginManager/releases,下载  Plugi ...

  2. mybatis基础，mybatis配置文件核心组件typeAliases元素

    typeAliases元素,术语类型别名 类型别名是为 Java 类型设置一个短的名字.它只和 XML 配置有关,存在的意义仅在于用来减少类完全限定名的冗余 <typeAliases> & ...

  3. std::vector<std::vector<> >

    上次看到这个有点晕了,其实这个vector保存的是std::vector<> #include <vector> #include <iostream> using ...

  4. .net framework 4.5 在Visual studio 2015中丢失

    解决办法:从另一台C:\Program Files(x86)\Reference Assemblies\Microsoft\.NetFramework 成功的环境中copy .net4.5 文件夹到错 ...

  5. SharePoint解决方案及开发系列(2)-ECM

    很多次跟客户做咨询及沟通的时候,客户都问SharePoint能做什么?是不是就是做文档管理?为什么要花那么多的钱没SharePoint?高大上? 我上家公司面试的时候,我的那个BOSS面试官有一个问题 ...

  6. 关东升的《从零开始学Swift》即将出版

    大家好: 苹果2015WWDC大会发布了Swift2.0,它较之前的版本Swift1.x有很大的变化,所以我即将出版<从零开始学Swift> <从零开始学Swift>将在< ...

  7. TGraphicControl（自绘就2步，直接自绘自己，不需要调用VCL框架提供的函数重绘所有子控件，也不需要自己来提供PaintWindow函数让管理框架来调用）与TControl关键属性方法速记（Repaint要求父控件执行详细代码来重绘自己，还是直接要求Invalidate无效后Update刷新父控件，就看透明不透明这个属性，因为计算显示的区域有所不同）

    TGraphicControl = class(TControl) private FCanvas: TCanvas; procedure WMPaint(var Message: TWMPaint) ...

  8. TCustomControl绘制自己和图形子控件共四步，TWinControl关键属性方法速记

    TCustomControl = class(TWinControl) private FCanvas: TCanvas; procedure WMPaint(var Message: TWMPain ...

  9. python多线程安全local()

    import time import threading from threading import local class Foo(local): pass # 实例化(创建)对象 foo = Fo ...

  10. Apache JServ Protocol (AJP)

    The Apache JServ Protocol (AJP) is a binary protocol that can proxy inbound requests from a web serv ...