有状态函数:独立于平台的有状态无服务器堆栈   这是一种在现代基础设施上创建高效.可扩展且一致的应用程序的简单方法,无论规模大小.   有状态函数是一种API,它通过为无服务器架构构建的运行时简化了分布式有状态应用程序的构建.它结合了有状态流处理的优点--处理延迟低.资源受限的大型数据集--以及为支持位置透明性.并发性.可伸缩性和弹性的有状态实体建模的运行时.     它旨在与现代体系结构(如云本机部署)和流行的事件驱动FaaS平台(如AWS Lambda和KNative)配合使用,并提供开箱即

综述: 在Flink中DataStream程序是在数据流上实现了转换的常规程序. 1.示范程序 import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streamin

第一章.flink实时数仓入门 一.依赖 <!--Licensed to the Apache Software Foundation (ASF) under oneor more contributor license agreements. See the NOTICE filedistributed with this work for additional informationregarding copyright ownership. The ASF licenses this fi

配置环境 包括 JAVA_HOME jobmanager.rpc.address jobmanager.heap.mb 和 taskmanager.heap.mb taskmanager.numberOfTaskSlots taskmanager.tmp.dirs slaves文件 启动关闭bin/start-cluster.shbin/stop-cluster.sh   初步使用 public static void main(String[] args) throws Exception {

Flink resource manager的作用如图,   FlinkResourceManager /** * * <h1>Worker allocation steps</h1> * * <ol> * <li>The resource manager decides to request more workers. This can happen in order * to fill the initial pool, or as a result o

Apache Flink闻名已久,一直没有亲自尝试一把,这两天看了文档,发现在real-time streaming方面,Flink提供了更多高阶的实用函数. 用Apache Flink实现WordCount 下载Apache Flink 0.10.1 启动local模式 bin/start-local.sh 运行scala-shell bin/start-scala-shell.sh remote localhost 6123 Flink中JobManager的默认监听端口是6123 word

InstanceManager用于管理JobManager申请到的taskManager和slots资源 /** * Simple manager that keeps track of which TaskManager are available and alive. */ public class InstanceManager { // ------------------------------------------------------------------------ //

  参考, http://wuchong.me/blog/2016/05/25/flink-internals-window-mechanism/ http://wuchong.me/blog/2016/06/06/flink-internals-session-window/    WindowOperator window operator通过WindowAssigner和Trigger来实现它的逻辑 当一个element到达时,通过KeySelector先assign一个key,并且通过W

org.apache.flink.streaming.api.windowing.triggers;   Trigger public abstract class Trigger<T, W extends Window> implements Serializable { /** * Called for every element that gets added to a pane. The result of this will determine * whether the pane

如果要考虑易用性和效率,使用rocksDB来替代普通内存的kv是有必要的 有了rocksdb,可以range查询,可以支持columnfamily,可以各种压缩 但是rocksdb本身是一个库,是跑在RocksDBStateBackend中的 所以taskmanager挂掉后,数据还是没了, 所以RocksDBStateBackend仍然需要类似HDFS这样的分布式存储来存储snapshot   kv state需要由rockdb来管理,这是和内存或file backend最大的不同 Abstr

在Flink – Checkpoint 没有描述了整个checkpoint的流程,但是对于如何生成snapshot和恢复snapshot的过程,并没有详细描述,这里补充   StreamOperator /** * Basic interface for stream operators. Implementers would implement one of * {@link org.apache.flink.streaming.api.operators.OneInputStreamOper

  public class StreamTaskState implements Serializable, Closeable { private static final long serialVersionUID = 1L; private StateHandle<?> operatorState; private StateHandle<Serializable> functionState; private HashMap<String, KvStateSnaps

Where did we come from? With the 0.9.0-milestone1 release, Apache Flink added an API to process relational data with SQL-like expressions called the Table API. The central concept of this API is a Table, a structured data set or stream on which relat

Flink在流上最大的特点,就是引入全局snapshot,   CheckpointCoordinator 做snapshot的核心组件为, CheckpointCoordinator /** * The checkpoint coordinator coordinates the distributed snapshots of operators and state. * It triggers the checkpoint by sending the messages to the re

  Metrics是以MetricsGroup来组织的 MetricGroup MetricGroup 这就是个metric容器,里面可以放subGroup,或者各种metric 所以主要的接口就是注册, /** * A MetricGroup is a named container for {@link Metric Metrics} and further metric subgroups. * * <p>Instances of this class can be used to re

1.概述 Flink 1.1.0 版本已经在官方发布了,官方博客于 2016-08-08 更新了 Flink 1.1.0 的变动.在这 Flink 版本的发布,添加了 SQL 语法这一特性.这对于业务场景复杂,依赖于 SQL 来分析统计数据,算得上是一个不错的福利.加上之前有同学和朋友邮件中提到,Flink 官方给的示例运行有困难,能否整合一下 Flink 的案例.笔者通过本篇博客来解答一下相关疑问. 2.内容 2.1 集群部署 首先,集群的部署需要 JDK 环境.下载 JDK 以及配置 JAV

1.概述 在如今数据爆炸的时代,企业的数据量与日俱增,大数据产品层出不穷.今天给大家分享一款产品—— Apache Flink,目前,已是 Apache 顶级项目之一.那么,接下来,笔者为大家介绍Flink 的相关内容. 2.内容 2.1 What's Flink Apache Flink 是一个面向分布式数据流处理和批量数据处理的开源计算平台,它能够基于同一个Flink运行时(Flink Runtime),提供支持流处理和批处理两种类型应用的功能.现有的开源计算方案,会把流处理和批处理作为两种

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals FLIP-1 : Fine Grained Recovery from Task Failures When a task fails during execution, Flink currently resets the entire execution graph and triggers complete re-execution f

  读accumlator JobManager 在job finish的时候会汇总accumulator的值, newJobStatus match { case JobStatus.FINISHED => try { val accumulatorResults = executionGraph.getAccumulatorsSerialized() val result = new SerializedJobExecutionResult( jobID, jobInfo.duration,

http://wuchong.me/blog/2016/04/26/flink-internals-how-to-handle-backpressure/ https://ci.apache.org/projects/flink/flink-docs-master/internals/back_pressure_monitoring.html   反压之所以会是需要的,因为当源端流量过大,而消费端无法及时消费时,可能会导致job crash:比如内存耗尽等: 其实通过合理控制资源的使用,JSto