flume在抽取MySQL数据到kafka时报错,如下

[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.sink.kafka.KafkaSink - Failed to publish events
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
[SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:)
at java.lang.Thread.run(Thread.java:)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity full, consider committing more frequently, increasing capacity, or increasing thread count
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:)
... more
MemoryChannel的事务的Put队列满
解决:将channel的capacity的值调大即可

a1.channels.ch-.capacity = 
注意:Flume配置每行后边不能加注释,否则会导致配置不生效

org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count的更多相关文章

  1. Put queue for MemoryTransaction of capacity 10000 full, consider committing more frequently, increasing capacity or increasing thread count flume capacity 时间数

    package com.test; import org.apache.http.*;import org.apache.http.entity.ContentType;import org.apac ...

  2. 一个Flume 异常(Put queue for MemoryTransaction of capacity 100 full)的排查和解决思路

    最近在做一个分布式调用链跟踪系统, 在两个地方采用了flume (我使用的flume版本是1.5.0-cdh5.4.4),一个是宿主系统 ,用flume agent进行日志搜集. 一个是从kafka拉 ...

  3. Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)

        前期博客 Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解) 问题详情 启动agent服务 [hadoop@master flume-1.7.0]$ ...

  4. Apache Flume 学习笔记

    # 从http://flume.apache.org/download.html 下载flume ############################################# # 概述: ...

  5. Apache Flume 1.7.0 发布,日志服务器

    Apache Flume 1.7.0 发布了,Flume 是一个分布式.可靠和高可用的服务,用于收集.聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型.这是一个可靠.容错的服务. 本次更 ...

  6. org.apache.flume.FlumeException: NettyAvroRpcClient { host: xxx.xxx.xxx.xxx, port: 41100 }: RPC

    2014-12-19 01:05:42,141 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.star ...

  7. Apache Flume日志收集系统简介

    Apache Flume是一个分布式.可靠.可用的系统,用于从大量不同的源有效地收集.聚合.移动大量日志数据进行集中式数据存储. Flume简介 Flume的核心是Agent,Agent中包含Sour ...

  8. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  9. Flafka: Apache Flume Meets Apache Kafka for Event Processing

    The new integration between Flume and Kafka offers sub-second-latency event processing without the n ...

随机推荐

  1. windows下 兼容Python2和Python3

    windows下同时安装了python2和python3时,都可以配置环境变量,如果在命令行里输入python命令,windows会去环境变量里寻找Python的安装位置,如果先找到pytoon2的, ...

  2. 【BZOJ2799】[Poi2012]Salaries 乱搞

    [BZOJ2799][Poi2012]Salaries Description 给出一棵n个结点的有根树,结点用正整数1~n编号.每个结点有一个1~n的正整数权值,不同结点的权值不相同,并且一个结点的 ...

  3. sparse matrix

    w https://en.wikipedia.org/wiki/Sparse_matrix 稀疏矩阵存储格式总结+存储效率对比:COO,CSR,DIA,ELL,HYB - Bin的专栏 - 博客园ht ...

  4. CalendarUtil 日期操作工具类

    版权声明:本文为博主原创文章,未经博主允许不得转载. [java] view plain copy import java.util.Calendar; import java.text.DateFo ...

  5. CF85D Sum of Medians

    CF85D Sum of Medians 题意翻译 一个集合,初始为空.现有三个操作: 1.add:向集合里加入数x,保证加入前集合中没有数x: 2.del:从集合中删除数x,保证删除前集合中有x: ...

  6. chrome浏览器自动填充失效问题

    现在浏览器带有自动填充,一般在input标签中增加autocomplete="off" 可以进行控制,off代表不填充,on代表填充,这个属性也可以放在form标签中,对所有的in ...

  7. Eclipse解决JSP页面乱码问题

    解决方案 1.首先选择 Window  -->  Preferences  -->  General-->  Content Types --> text (点击左边的+,显示 ...

  8. 1.6 使用电脑测试MC20的读取带中文短信功能

    需要准备的硬件 MC20开发板 1个 https://item.taobao.com/item.htm?id=562661881042 GSM/GPRS天线 1根 https://item.taoba ...

  9. Apache Shiro:【2】与SpringBoot集成完成登录验证

    Apache Shiro:[2]与SpringBoot集成完成登录验证 官方Shiro文档:http://shiro.apache.org/documentation.html Shiro自定义Rea ...

  10. hadoop linux 杂记

    切换到root        su    修改sudo sudo + 命令 --> root权限 + 命令        su root        vim /etc/sudoers      ...