three supported reliability levels: * End-to-end * Store on failure * Best effort
https://github.com/cloudera/flume/blob/master/flume-docs/src/docs/UserGuide/Introduction
| === Reliability | |
| Reliability, the ability to continue delivering events in the face of | |
| failures without losing data, is a vital feature of Flume. Large | |
| distributed systems can and do suffer partial failures in many ways - | |
| physical hardware can fail, resources such as network bandwidth or | |
| memory can become scarce, or software can crash or run slowly. Flume | |
| emphasizes fault-tolerance as a core design principle and keeps | |
| running and collecting data even when many components have failed. | |
| Flume can guarantee that all data received by an agent node will | |
| eventually make it to the collector at the end of its flow as long as | |
| the agent node keeps running. That is, data can be *reliably* | |
| delivered to its eventual destination. | |
| However, reliable delivery can be very resource intensive and is often | |
| a stronger guarantee than some data sources require. Therefore, Flume | |
| allows the user to specify, on a per-flow basis, the level of | |
| reliability required. There are three supported reliability levels: | |
| * End-to-end | |
| * Store on failure | |
| * Best effort | |
| .A Note About Reliability | |
| ****************** | |
| Although Flume is extremely tolerant to machine, network, and software | |
| failures, there is never any such thing as '100% reliability'. If all | |
| the machines in a Flume installation were irrevocably destroyed in | |
| some terrible data center incident, all copies of Flume's data would | |
| be lost and there would be no way to recover them. Therefore all of | |
| Flume's reliability levels make guarantees about data delivery 'until | |
| some maximum number of failures have occurred'. Flume's failure modes | |
| - in terms of what can fail and what will keep running if they do - | |
| are described in detail later in this guide. | |
| ****************** | |
| The *end-to-end* reliability level guarantees that once Flume accepts | |
| an event, that event will make it to the endpoint - as long as the | |
| agent that accepted the event remains live long enough. The first | |
| thing the agent does in this setting is write the event to disk in a | |
| ''write-ahead log'' (WAL) so that, if the agent crashes and restarts, | |
| knowledge of the event is not lost. After the event has successfully | |
| made its way to the end of its flow, an acknowledgment is sent back to | |
| the originating agent so that it knows it no longer needs to store the | |
| event on disk. This reliability level can withstand any number of | |
| failures downstream of the initial agent. | |
| The *store on failure* reliability level causes nodes to only require | |
| an acknowledgement from the node one hop downstream. If the sending | |
| node detects a failure, it will store data on its local disk until the | |
| downstream node is repaired, or an alternate downstream destination | |
| can be selected. While this is effective, data can be lost if a | |
| compound or silent failure occurs. | |
| The *best-effort* reliability level sends data to the next hop with no | |
| attempts to confirm or retry delivery. If nodes fail, any data that | |
| they were in the process of transmitting or receiving can be | |
| lost. This is the weakest reliability level, but also the most | |
| lightweight. |
| === Reliability | |
| Reliability, the ability to continue delivering events in the face of | |
| failures without losing data, is a vital feature of Flume. Large | |
| distributed systems can and do suffer partial failures in many ways - | |
| physical hardware can fail, resources such as network bandwidth or | |
| memory can become scarce, or software can crash or run slowly. Flume | |
| emphasizes fault-tolerance as a core design principle and keeps | |
| running and collecting data even when many components have failed. | |
| Flume can guarantee that all data received by an agent node will | |
| eventually make it to the collector at the end of its flow as long as | |
| the agent node keeps running. That is, data can be *reliably* | |
| delivered to its eventual destination. | |
| However, reliable delivery can be very resource intensive and is often | |
| a stronger guarantee than some data sources require. Therefore, Flume | |
| allows the user to specify, on a per-flow basis, the level of | |
| reliability required. There are three supported reliability levels: | |
| * End-to-end | |
| * Store on failure | |
| * Best effort | |
| .A Note About Reliability | |
| ****************** | |
| Although Flume is extremely tolerant to machine, network, and software | |
| failures, there is never any such thing as '100% reliability'. If all | |
| the machines in a Flume installation were irrevocably destroyed in | |
| some terrible data center incident, all copies of Flume's data would | |
| be lost and there would be no way to recover them. Therefore all of | |
| Flume's reliability levels make guarantees about data delivery 'until | |
| some maximum number of failures have occurred'. Flume's failure modes | |
| - in terms of what can fail and what will keep running if they do - | |
| are described in detail later in this guide. | |
| ****************** | |
| The *end-to-end* reliability level guarantees that once Flume accepts | |
| an event, that event will make it to the endpoint - as long as the | |
| agent that accepted the event remains live long enough. The first | |
| thing the agent does in this setting is write the event to disk in a | |
| ''write-ahead log'' (WAL) so that, if the agent crashes and restarts, | |
| knowledge of the event is not lost. After the event has successfully | |
| made its way to the end of its flow, an acknowledgment is sent back to | |
| the originating agent so that it knows it no longer needs to store the | |
| event on disk. This reliability level can withstand any number of | |
| failures downstream of the initial agent. | |
| The *store on failure* reliability level causes nodes to only require | |
| an acknowledgement from the node one hop downstream. If the sending | |
| node detects a failure, it will store data on its local disk until the | |
| downstream node is repaired, or an alternate downstream destination | |
| can be selected. While this is effective, data can be lost if a | |
| compound or silent failure occurs. | |
| The *best-effort* reliability level sends data to the next hop with no | |
| attempts to confirm or retry delivery. If nodes fail, any data that | |
| they were in the process of transmitting or receiving can be | |
| lost. This is the weakest reliability level, but also the most | |
| lightweight. |
three supported reliability levels: * End-to-end * Store on failure * Best effort的更多相关文章
- SignalR Supported Platforms -摘自网络
SignalR is supported under a variety of server and client configurations. In addition, each transpor ...
- store操作
store.remove(rs); store.sync({ success: function (e, opt) { this.store.commitChanges(); }, failure: ...
- extjs 解决使用store.sync()方法更新item有时不触发后台action的问题
问题描述: extjs 解决使用store.sync()方法更新item有时不触发后台action,不出发后台action的原因是item的字段值没有变化 解决方法: item.setDirty(tr ...
- PMP用语集
AC actual cost 实际成本 ACWP actual cost of work performed 已完工作实际成本 BAC budget at completion 完工预算 BCWP b ...
- Solaris10安装配置LDAP(iPlanet Directory Server )
Solaris10安装光盘自带了iPlanet Directory Server安装包,系统管理员可以利用iPlanet Directory Server在Solaris系统创建一个LDAP Serv ...
- memory ordering 内存排序
Memory ordering - Wikipedia https://en.wikipedia.org/wiki/Memory_ordering https://zh.wikipedia.org/w ...
- Web测试介绍2一 安全测试
安全测试是在IT软件产品的生命周期中,特别是产品开发基本完成到发布阶段,对产品进行检验以验证产品符合安全需求定义和产品质量标准的过程. 主要安全需求包括: (i) 认证 Authent ...
- Python 3.6.0的sqlite3模块无法执行VACUUM语句
Python 3.6.0的sqlite3模块存在一个bug(见issue 29003),无法执行VACUUM语句. 一执行就出现异常: Traceback (most recent call last ...
- Flume1.5.0的安装、部署、简单应用(含伪分布式、与hadoop2.2.0、hbase0.96的案例)
目录: 一.什么是Flume? 1)flume的特点 2)flume的可靠性 3)flume的可恢复性 4)flume 的 一些核心概念 二.flume的官方网站在哪里? 三.在哪里下载? 四.如何安 ...
随机推荐
- anaconda安装及环境变量配置
最近无聊想的学习一下python. 首先可以在官网上下载与自己电脑兼容的anaconda安装包,网址如下https://www.continuum.io/downloads 下载好后就是一个exe文件 ...
- 最全py2exe
这次不是直接讲解下去,而是谈一下如何把我们写的游戏做成一个exe文件,这样一来,用户不需要安装python就可以玩了.扫清了游戏发布一大障碍啊! perl,python,java等编程语言,非常好用, ...
- checkbox 复选框只能选中一次,之后不能用
<td> <input type="checkbox" name="is_check" id="is_check" val ...
- dedecms--二次开发之前后台登录分开
最近在写dedecms系统下会员功能二次开发,然后发现在本地测试的时候每次登录后台,管理员帐号都会在前台页面也显示登录了,但是如果真的是在前台页面用管理员账号登录的话那是登陆不了的,所以我觉得这样的效 ...
- MinGW 创建的程序或 DLL 脱离 libgcc-xx-xx.dll 和 libstdc++-x.dll 运行库的方法
MinGW 沿袭了 Linux 下 gcc/g++ 的习惯,编译出的程序或者动态链接库(共享库)总是默认采用动态链接方式,需要系统中附带运行时库文件 libgcc-xx-xx.dll 和 libstd ...
- AC日记——小行星 洛谷 P2711
题目背景 pid=3437 题目描述 星云中有n颗行星,每颗行星的位置是(x,y,z).每次可以消除一个面(即x,y或z坐标相等)的行星,但是由于时间有限,求消除这些行星的最少次数. 输入输出格式 输 ...
- Codeforces635C XOR Equation【数学】
题目链接: http://codeforces.com/contest/635/problem/C 题意: 给定两个数的和s及异或x,求两个数的可能情况. 分析: 我们有公式a+b=a& b∗ ...
- Swoole RPC 的实现
目录 概述 实现效果 代码 小结 概述 这是关于 Swoole 学习的第七篇文章:Swoole RPC 的实现. 第六篇:Swoole 整合成一个小框架 第五篇:Swoole 多协议 多端口 的应用 ...
- T2821 天使之城 codevs
http://codevs.cn/problem/2821/ 题目描述 Description 天使城有一个火车站,每辆火车都从A方向驶入车站,再从B方向驶出车站. 为了调度火车,火车站设有停放轨道, ...
- FMDB支持的事务类型
FMDB支持的事务类型 在数据库中,事务可以保证数据操作的完整性.当存在大量并发操作,容易出现死锁问题.在SQLite中,为了解决该问题,提供三种事务模式,分别为DEFFERED.IMMEDIAT ...