flume在抽取MySQL数据到kafka时报错，如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: Failed to publish events

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)

    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)

    at java.lang.Thread.run(Thread.java:)

Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity  full, consider committing more frequently, increasing capacity, or increasing thread count

    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)

    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)

    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)

    at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)

    ...  more


MemoryChannel的事务的Put队列满

解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity = 
注意：Flume配置每行后边不能加注释，否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

  1. Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数

    package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...

  2. 一个Flume 异常（Put queue for MemoryTransaction of capacity 100 full）的排查和解决思路

    最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...

  3. Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法（图文详解）

        前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...

  4. Apache Flume 学习笔记

    # 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...

  5. Apache Flume 1.7.0 发布，日志服务器

    Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...

  6. org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC

    2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...

  7. Apache Flume日志收集系统简介

    Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...

  8. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  9. Flafka: Apache Flume Meets Apache Kafka for Event Processing

    The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

  1. Javascript中的函数中的this值

    看下面这段代码会在控制台上输出什么内容? <script> var url="fang.com"; var obj={ url:"soufun.com&quo ...

  2. Cocos2d-x Lua中生命周期函数

    场景(Scene)以及所有节点(Node)的生命周期事件如下:enter.进入场景时候触发.enterTransitionFinish.进入场景而且过渡动画结束时候触发.exit.退出场景时候触发 . ...

  3. mysql编译参数详解（./configure）

    1.--prefix=PREFIX:指定程序安装路径: 2.--enable-assembler:使用汇编模式:(文档说明:compiling in x86 (and sparc) versions  ...

  4. 巨蟒python全栈开发linux之cento8

    1.复习路飞部署学习 2.mysql数据库远程访问 3.mysql主从复制 4.redis安全方式启动 5.redis数据类型 6.redis发布订阅 7.redis持久化 8.redis哨兵 9.r ...

  5. 巨蟒python全栈开发django7:多表增加和查询

    1.回顾内容&&补充 补充1: 补充2: 这个选择的是第二个解释器. 选择第一个的话,只是针对当前的项目,如果再开新项目的话,需要重新下载安装相关的包. 点击保存,因为我们注释掉了,创 ...

  6. 明文post密码

    w 作者:余天升链接:https://www.zhihu.com/question/20306241/answer/14696464 看到上面几位的回答,我真心觉得,当前信息安全保护的意识过于低下,连 ...

  7. Java内存模型之happens-before

    转载自    http://www.cnblogs.com/chenssy/p/6393321.html 无论处理器.JVM.编译器都会都保证程序正确的前提下尽可能的对指令执行效率进行优化,进行指令重 ...

  8. Asp.Net MVC anti-forgery token的问题：nameidentifier or identityprovider not present

    当使用ClaimsIdentity的时候,Asp.Net MVC在生成AntiForgeryToken的时候会默认使用User.Identity中两种ClaimsType的值:NameIdentifi ...

  9. python——列表&字符串互相转换方法小结

    字符串(str)转列表(list) 转换方法:str.split() str = 'zhu gao chao' print(str.split(' ')) # 用split进行转换 str——> ...

  10. Android系统移植与调试之------->如何修改Android设备存储盘符名称与Android设备的型号

    一.修改Android设备存储盘符名称 (注:TBDG1073为我的项目名称) 1.修改device/other/TBDG1073/system.prop 文件 2.修改ro.media.patiti ...