Apache Flume 1.7.0 发布了,Flume 是一个分布式、可靠和高可用的服务,用于收集、聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型。这是一个可靠、容错的服务。

本次更新如下:

新特性

  • [FLUME-2498] - Implement Taildir Source

改进

  • [FLUME-1899] - Make SpoolDir work with Sub-Directories

  • [FLUME-2526] - Build flume by jdk 7 in default

  • [FLUME-2628] - Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source

  • [FLUME-2704] - Configurable poll delay for spooling directory source

  • [FLUME-2718] - HTTP Source to support generic Stream Handler

  • [FLUME-2729] - Allow pollableSource backoff times to be configurable

  • [FLUME-2755] - Kafka Source reading multiple topics

  • [FLUME-2781] - A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source

  • [FLUME-2799] - Kafka Source - Message Offset and Partition add to headers

  • [FLUME-2801] - Performance improvement on TailDir source

  • [FLUME-2810] - Add static Schema URL to AvroEventSerializer configuration

  • [FLUME-2820] - Support New Kafka APIs

  • [FLUME-2852] - Kafka Source/Sink should optionally read/write Flume records

  • [FLUME-2868] - Kafka Channel partition topic by key

  • [FLUME-2872] - Kafka Sink should be able to select which header as the key

  • [FLUME-2875] - Allow RollingFileSink to specify a file prefix and a file extension.

  • [FLUME-2909] - Bump Rat version

  • [FLUME-2910] - AsyncHBaseSink - Failure callbacks should log the exception that caused them

  • [FLUME-2911] - Add includePattern option in SpoolDirectorySource configuration

  • [FLUME-2918] - TaildirSource is underperforming with huge parent directories

  • [FLUME-2937] - Integrate checkstyle for non-test classes

  • [FLUME-2941] - Integrate checkstyle for test classes

  • [FLUME-2954] - make raw data appearing in log messages explicit

  • [FLUME-2955] - Add file path to the header in TaildirSource

  • [FLUME-2959] - Fix issues with flume-checkstyle module

  • [FLUME-2982] - Add localhost escape sequence to HDFS sink

  • [FLUME-2999] - Kafka channel and sink should enable statically assigned partition per event via header

  • [FLUME-2821] - Flume-Kafka Source with new Consumer

  • [FLUME-2822] - Flume-Kafka-Sink with new Producer

  • [FLUME-2823] - Flume-Kafka-Channel with new APIs

Bug 修复

  • [FLUME-1668] - Hdfs Sink File Rollover

  • [FLUME-2132] - Exception while syncing from Flume to HDFS

  • [FLUME-2143] - Flume build occasionally fails with OutOfMemoryError on Windows.

  • [FLUME-2215] - ResettableFileInputStream can’t support ucs-4 character

  • [FLUME-2318] - SpoolingDirectory is unable to handle empty files

  • [FLUME-2448] - Building flume from trunk failing with dependency error

  • [FLUME-2484] - NullPointerException in Kafka Sink test

  • [FLUME-2485] - Thrift Source tests fail on Oracle JDK 8

  • [FLUME-2514] - Some TestFileChannelRestart tests are extremely slow

  • [FLUME-2567] - Remove unneeded repository declarations in pom.xml

  • [FLUME-2573] - flume-ng –conf parameter is not used when starting a flume agent

  • [FLUME-2593] - ResettableFileInputStream returns negate values from read() method

  • [FLUME-2619] - Spooldir source does not log channel exceptions

  • [FLUME-2632] - High CPU on KafkaSink

  • [FLUME-2652] - Documented transaction handling semantics incorrect

  • [FLUME-2660] - Add documentation for EventValidator

  • [FLUME-2672] - NPE in KafkaSourceCounter

  • [FLUME-2712] - Optional channel errors slows down the Source to Main channel event rate

  • [FLUME-2725] - HDFS Sink does not use configured timezone for rounding

  • [FLUME-2732] - Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable

  • [FLUME-2734] - Kafka Channel timeout property is overridden by default value

  • [FLUME-2738] - Async HBase sink FD leak on client shutdown

  • [FLUME-2746] - How to include this Flume Patch in Flume 1.5.2 ?

  • [FLUME-2749] - Kerberos configuration error when using short names in multiple HDFS Sinks

  • [FLUME-2751] - Upgrade Derby version to 10.11.1.1

  • [FLUME-2753] - Error when specifying empty replace string in Search and Replace Interceptor

  • [FLUME-2754] - Hive Sink skipping first transaction in each Batch of Hive Transactions

  • [FLUME-2761] - Move Hive sink out of preview mode

  • [FLUME-2763] - flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib

  • [FLUME-2773] - TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already

  • [FLUME-2797] - SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource

  • [FLUME-2798] - Malformed Syslog messages can lead to OutOfMemoryException

  • [FLUME-2804] - Hive sink - abort remaining transactions on shutdown

  • [FLUME-2806] - flume-ng.ps1 Error running script to start an agent on Windows

  • [FLUME-2835] - Hive Sink tests need to create table with transactional property set

  • [FLUME-2841] - Upgrade commons-collections to 3.2.2

  • [FLUME-2844] - ChannelCounter of SpillableMemoryChannel doesn’t register actually.

  • [FLUME-2881] - Windows Launch Script fails in plugins dir code

  • [FLUME-2886] - Optional Channels can cause OOMs

  • [FLUME-2889] - Fixes to DateTime computations

  • [FLUME-2891] - Revert FLUME-2712 and FLUME-2886

  • [FLUME-2897] - AsyncHBase sink NPE when Channel.getTransaction() fails

  • [FLUME-2901] - Document Kerberos setup for Kafka channel

  • [FLUME-2908] - NetcatSource - SocketChannel not closed when session is broken

  • [FLUME-2913] - Flume classpath too long

  • [FLUME-2915] - The kafka channel using new APIs will be stuck when the sink is avro sink

  • [FLUME-2920] - Kafka Channel Should Not Commit Offsets When Stopping

  • [FLUME-2922] - HDFSSequenceFile Should Sync Writer

  • [FLUME-2923] - Bump AsyncHBase version

  • [FLUME-2936] - KafkaSource tests arbitrarily fail

  • [FLUME-2939] - Upgrade recursive SpoolDir to use Java7 features

  • [FLUME-2948] - Docs: Fixed parameters on Replicating Channel Selector documentation example

  • [FLUME-2949] - Flume fails to build on Windows

  • [FLUME-2950] - ReliableSpoolingFileEventReader.rollCurrentFile is broken

  • [FLUME-2952] - SyslogAgent possible NPE on stop()

  • [FLUME-2972] - Handle offset migration in the new Kafka Channel

  • [FLUME-2974] - Some tests are broken in TestReliableSpoolingFileEventReader and TestSpoolingFileLineReader

  • [FLUME-2983] - Handle offset migration in the new Kafka Source

文档

  • [FLUME-2575] - FLUME-2548 brings SSLv2Hello back for Avro Sink, but UG says it is one of the protocols to exclude

  • [FLUME-2713] - Document Fault Tolerant Config parameters in FlumeUserGuide

  • [FLUME-2737] - Documentation for Pollable Source config parameters introduced in FLUME-2729

  • [FLUME-2783] - Update Website Team page with new Committer’s

  • [FLUME-2890] - Typo in Twitter source warning 悦德财富:https://www.yuedecaifu.com

  • [FLUME-2934] - Document new cachePatternMatching option for TaildirSource

  • [FLUME-2963] - FlumeUserGuide - error in Kafka Source properties table

  • [FLUME-2971] - Document secure Kafka Sink/Source/Channel setup

  • [FLUME-2975] - Minor mistake in NetCat Source example in documentation

  • [FLUME-2998] - Add missing configuration parameter to SequenceSource docs

任务

  • [FLUME-2935] - Bump java target version to 1.7

测试

  • [FLUME-3003] - testSourceCounter in TestSyslogUdpSource is flaky

Apache Flume 1.7.0 发布,日志服务器的更多相关文章

  1. Apache Flume 1.6.0 发布,日志服务器

    Apache Flume 1.6.0 发布,此版本现已提供下载: http://flume.apache.org/download.html 更新日志和文档: http://flume.apache. ...

  2. Ajenti 1.0 发布,服务器管理系统 - 开源中国社区

    Ajenti 1.0 发布,服务器管理系统 - 开源中国社区 Ajenti 1.0 发布,服务器管理系统

  3. Apache Flume 1.7.0 源码编译 导入Eclipse

    前言 最近看了看Apache Flume,在虚拟机里跑了一下flume + kafka + storm + mysql架构的demo,功能很简单,主要是用flume收集数据源(http上报信息),放入 ...

  4. Visual Studio的 Apache Cordova 插件CTP3.0发布!

    北京时间12号晚23点开始的Connect()活动上,微软发布了一系列激动人心的消息! .NET开源了!以后.NET将可在Linux和Mac OS平台上运行! VS免费了!!如果你是学生,个人开发者, ...

  5. Apache Flume 1.7.0 各个模块简介

    Flume简介 Apache Flume是一个分布式.可靠.高可用的日志收集系统,支持各种各样的数据来源,如http,log文件,jms,监听端口数据等等,能将这些数据源的海量日志数据进行高效收集.聚 ...

  6. Apache Storm 1.1.0 发布概览

    写在前面的话 本人长期关注数据挖掘与机器学习相关前沿研究.欢迎和我交流,私人微信:846731084 我自己测试了一下这个版本,总的来说更加稳定,新增的特性并没有一一测试,仅凭kafk-client来 ...

  7. Apache Flume 安装文档、日志收集

    简介: 官网 http://flume.apache.org 文档 https://flume.apache.org/FlumeUserGuide.html hadoop 生态系统中,flume 的职 ...

  8. Apache Flume 1.7.0 自定义输入输出

    自定义http source config a1.sources.r1.type=http a1.sources.r1.bind=localhost a1.sources.r1.port= a1.so ...

  9. 当Atlas遇见Flink——Apache Atlas 2.2.0发布!

    距离上次atlas发布新版本已经有一年的时间了,但是这一年元数据管理平台的发展一直没有停止.Datahub,Amundsen等等,都在不断的更新着自己的版本.但是似乎Atlas在元数据管理,数据血缘领 ...

随机推荐

  1. 调试多线程 & 查死锁的bug & gcore命令 & gdb对多线程的调试 & gcore & pstack & 调试常用命令

    gdb thread apply all bt 如果你发现有那么几个栈停在 pthread_wait 或者类似调用上,大致就可以得出结论:就是它们几个儿女情长,耽误了整个进程. 注意gdb的版本要高于 ...

  2. C语言局部变量和全局变量的区别。——Arvin

    局变量是使用相同的内存块在整个类中存储一个值. 全局变量的存在主要有以下一些原因:  1,使用全局变量会占用更多的内存(因为其生命期长),不过在计算机配置很高的今天,这个不成为什么问题,除非使用的是巨 ...

  3. ATC空管系统的实时控制软件系统分析

    什么是ATC空管系统? 空中交通管制的目的是对航空器的空中活动进行有效的管理,维护空中交通秩序,保障空中交通畅通,保证飞行安全和提高飞行效率,防止航空器相撞,防止机场及其附近空域的航空器同障碍物相撞. ...

  4. excel读取

    一.jar包 二.工具类 package excel; import java.io.FileInputStream; import java.io.FileNotFoundException; im ...

  5. selenium移动div里面的滚动条,操作弹出对话框

    还是使用js来移动 首先要定位到这个元素 倾向于使用js来定位元素,输入下面的脚本,按下回车键,即可在调试页面看到对应的div块$("div.table-responsive") ...

  6. 关于absolute 和 relative 定位的定义

    absolute的英文意思是绝对的意思,实际上是针对父级元素元素定位,如果父级元素没有position:relative|absolute,则追至再上一个父级元素,直至相对于文档的左上角定位,按照我们 ...

  7. 网站指纹识别工具——WhatWeb v0.4.7发布

      WhatWeb是一款网站指纹识别工具,主要针对的问题是:“这个网站使用的什么技术?”WhatWeb可以告诉你网站搭建使用的程序,包括何种CMS系统.什么博客系统.Javascript库.web服务 ...

  8. F:ungeon Master

    总时间限制: 1000ms 内存限制: 65536kB描述You are trapped in a 3D dungeon and need to find the quickest way out! ...

  9. linux环境下android-ndk下的ffmpeg编译

    目前正在做手机底层播放器对的开发,需要用的ffmpeg,因为是新手,所以先从ffmpeg的编译开始做起.虽然是在前人的基础上,但是在linux上编译确实头一遭,因此在编译中,总有些坑是必须要填的,下面 ...

  10. 开发Blog整理

    开发Blog记录 清理收藏夹 太多了,来不及看了. http://blog.sina.com.cn/s/blog_67d95f40010113ec.htmlhttp://segmentfault.co ...