Structured Streaming Programming Guide
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
http://www.slideshare.net/databricks/a-deep-dive-into-structured-streaming
Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.
You can express your streaming computation the same way you would express a batch computation on static data.
The Spark SQL engine will take care of running it incrementally and continuously and updating the final result as streaming data continues to arrive. You can use the Dataset/DataFrame API in Scala, Java or Python to express streaming aggregations, event-time windows, stream-to-batch joins, etc. The computation is executed on the same optimized Spark SQL engine.
Finally, the system ensures end-to-end exactly-once fault-tolerance guarantees through checkpointing and Write Ahead Logs.
In short, Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing without the user having to reason about streaming.
你可以像在静态数据源上一样,使用DataFrame接口去执行SQL,这些SQL会跑在和batch相同的optimized Spark SQL engine上
并且可以保证exactly-once fault-tolerance,通过checkpointing and Write Ahead Logs
只是将DStream抽象,换成DataFrame,即table
这样就可以进行结构化的操作,
并且基本和处理batch数据一样,
可以看到差别不大
整个过程是这样的,
可以看到,这里的output模式是complete,因为有聚合,所以每次输出需要,输出until now的统计数据
输出的mode,分为,
The “Output” is defined as what gets written out to the external storage. The output can be defined in different modes
Complete Mode - The entire updated Result Table will be written to the external storage. It is up to the storage connector to decide how to handle writing of the entire table.
Append Mode - Only the new rows appended in the Result Table since the last trigger will be written to the external storage. This is applicable only on the queries where existing rows in the Result Table are not expected to change.
Update Mode - Only the rows that were updated in the Result Table since the last trigger will be written to the external storage (not available yet in Spark 2.0). Note that this is different from the Complete Mode in that this mode does not output the rows that are not changed.
complete mode上面的例子已经给出
append mode,就是每次只输出增量,这个对于没有聚合的场景就是合适的
Window Operations on Event Time
spark认为自己对于Event time是天然支持的,只需要把它作为dataframe里面的一个列,然后做groupby即可以
然后对于late data,因为是增量输出的,所以也是可以handle的
Fault Tolerance Semantics
Delivering end-to-end exactly-once semantics was one of key goals behind the design of Structured Streaming.
To achieve that, we have designed the Structured Streaming sources, the sinks and the execution engine to reliably track the exact progress of the processing so that it can handle any kind of failure by restarting and/or reprocessing. Every streaming source is assumed to have offsets (similar to Kafka offsets, or Kinesis sequence numbers) to track the read position in the stream. The engine uses checkpointing and write ahead logs to record the offset range of the data being processed in each trigger. The streaming sinks are designed to be idempotent for handling reprocessing. Together, using replayable sources and idempotant sinks, Structured Streaming can ensure end-to-end exactly-once semantics under any failure.
首先依赖source是可以依据offset replay,而sink是幂等的,这样只需要通过Write Ahead Logs记录offset,checkpoint记录state,就可以做到exactly once,因为本质是batch
Structured Streaming Programming Guide的更多相关文章
- Structured Streaming Programming Guide结构化流编程指南
目录 Overview Quick Example Programming Model Basic Concepts Handling Event-time and Late Data Fault T ...
- Spark Streaming Programming Guide
参考,http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html Overview SparkStre ...
- spark第六篇:Spark Streaming Programming Guide
预览 Spark Streaming是Spark核心API的扩展,支持高扩展,高吞吐量,实时数据流的容错流处理.数据可以从Kafka,Flume或TCP socket等许多来源获取,并且可以使用复杂的 ...
- Spark Structured streaming框架(1)之基本使用
Spark Struntured Streaming是Spark 2.1.0版本后新增加的流计算引擎,本博将通过几篇博文详细介绍这个框架.这篇是介绍Spark Structured Streamin ...
- Spark Structured Streaming框架(2)之数据输入源详解
Spark Structured Streaming目前的2.1.0版本只支持输入源:File.kafka和socket. 1. Socket Socket方式是最简单的数据输入源,如Quick ex ...
- Spark Structured Streaming框架(5)之进程管理
Structured Streaming提供一些API来管理Streaming对象.用户可以通过这些API来手动管理已经启动的Streaming,保证在系统中的Streaming有序执行. 1. St ...
- Spark Structured Streaming框架(4)之窗口管理详解
1. 结构 1.1 概述 Structured Streaming组件滑动窗口功能由三个参数决定其功能:窗口时间.滑动步长和触发时间. 窗口时间:是指确定数据操作的长度: 滑动步长:是指窗口每次向前移 ...
- Spark Structured Streaming框架(3)之数据输出源详解
Spark Structured streaming API支持的输出源有:Console.Memory.File和Foreach.其中Console在前两篇博文中已有详述,而Memory使用非常简单 ...
- Spark Structured Streaming框架(2)之数据输入源详解
Spark Structured Streaming目前的2.1.0版本只支持输入源:File.kafka和socket. 1. Socket Socket方式是最简单的数据输入源,如Quick ex ...
随机推荐
- hdu 4622 **
题意:Suppose there are the symbols M, I, and U which can be combined to produce strings of symbols cal ...
- strcat函数造成的段错误(Segmentation fault)
转自:http://book.51cto.com/art/201311/419441.htm 3.21 strcat函数造成的段错误 代码示例 int main() { char dest[7]=& ...
- js:数据结构笔记10--图和图算法
图:是由边和定点的集合组成: 按照图的定点对是否有序可以分为:有向图和无向图: 路径:所有顶点都由边连接构成:路径长度为第一个定点到最后一个顶点之间的数量: 环:指向自身的顶点,长度为0:圈:至 ...
- http://www.cnblogs.com/yjmyzz/p/3941043.html
http://www.cnblogs.com/yjmyzz/p/3941043.html
- LightOJ1025 The Specials Menu(区间DP)
给一个字符串,问有几种删字符的方式使删后的非空字符串是个回文串. 当然区间DP:dp[i][j]表示子串stri...strj的方案数 感觉不好转移,可能重复算了.我手算了"AAA" ...
- 不容易系列之二[HDU2042]
不容易系列之二 Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/Others)Total Sub ...
- python 中chr(),unichr(),ord()的用法
chr()根据整数返回对应的字符,也就是讲ascii转换为字符 unichr()将整数返回成unicode字符 ord()将字符转换成ascii码
- POJ 3691 DNA repair(AC自动机+DP)
题目链接 能AC还是很开心的...此题没有POJ2778那么难,那个题还需要矩阵乘法,两个题有点相似的. 做题之前,把2778代码重新看了一下,回忆一下当时做题的思路,回忆AC自动机是干嘛的... 状 ...
- 深入理解javascript中的立即执行函数(function(){…})()
投稿:junjie 字体:[增加 减小] 类型:转载 时间:2014-06-12 我要评论 这篇文章主要介绍了深入理解javascript中的立即执行函数,立即执行函数也叫立即调用函数,通常它的写法是 ...
- SecureCrt脚本(一)顶级对象之Crt
Crt自动化 测试 SecureCrt脚本 JS脚本 1.引言 2.关于脚本表头 3.顶级对象'crt'的子属性和方法 3.1.属性 3.1.1.Dialog 3.1.2.Screen 3.1.3 ...