Flink的keyby延时源码
public class RecordWriter<T extends IOReadableWritable> {
==FullBuffer
/**
* This is used to send LatencyMarks to a random target channel.
*/
public void randomEmit(T record) throws IOException, InterruptedException {
sendToTarget(record, rng.nextInt(numChannels));
} private void sendToTarget(T record, int targetChannel) throws IOException, InterruptedException {
RecordSerializer<T> serializer = serializers[targetChannel]; SerializationResult result = serializer.addRecord(record); while (result.isFullBuffer()) {
if (tryFinishCurrentBufferBuilder(targetChannel, serializer)) {
// If this was a full record, we are done. Not breaking
// out of the loop at this point will lead to another
// buffer request before breaking out (that would not be
// a problem per se, but it can lead to stalls in the
// pipeline).
if (result.isFullRecord()) {
break;
}
}
BufferBuilder bufferBuilder = requestNewBufferBuilder(targetChannel); result = serializer.continueWritingWithNextBufferBuilder(bufferBuilder);
}
checkState(!serializer.hasSerializedData(), "All data should be written at once"); if (flushAlways) {
targetPartition.flush(targetChannel);
}
} ==getBufferTimeout
package org.apache.flink.streaming.runtime.tasks;
public abstract class StreamTask<OUT, OP extends StreamOperator<OUT>>
@VisibleForTesting
public static <OUT> List<StreamRecordWriter<SerializationDelegate<StreamRecord<OUT>>>> createStreamRecordWriters(
StreamConfig configuration,
Environment environment) {
List<StreamRecordWriter<SerializationDelegate<StreamRecord<OUT>>>> streamRecordWriters = new ArrayList<>();
List<StreamEdge> outEdgesInOrder = configuration.getOutEdgesInOrder(environment.getUserClassLoader());
Map<Integer, StreamConfig> chainedConfigs = configuration.getTransitiveChainedTaskConfigsWithSelf(environment.getUserClassLoader()); for (int i = 0; i < outEdgesInOrder.size(); i++) {
StreamEdge edge = outEdgesInOrder.get(i);
streamRecordWriters.add(
createStreamRecordWriter(
edge,
i,
environment,
environment.getTaskInfo().getTaskName(),
chainedConfigs.get(edge.getSourceId()).getBufferTimeout()));
}
return streamRecordWriters; } http://vinoyang.com/2016/12/30/flink-runtime-producer-result-partition/
http://vinoyang.com/2017/01/04/flink-runtime-consumer-input-gate/
http://vinoyang.com/2017/01/08/flink-runtime-netty-part-1/
http://vinoyang.com/2017/01/12/flink-runtime-netty-part-2/
http://vinoyang.com/2017/01/15/flink-runtime-netty-part-3/
http://vinoyang.com/2016/12/14/flink-runtime-NetworkEnvironment/
http://vinoyang.com/2016/12/28/flink-runtime-communicate-api/
http://vinoyang.com/archives/2016/12/
http://vinoyang.com/2016/12/20/flink-runtime-unified-data-exchange/
Flink的keyby延时源码的更多相关文章
- flink on yarn部分源码解析
转发请注明原创地址:https://www.cnblogs.com/dongxiao-yang/p/9403427.html flink任务的deploy形式有很多种选择,常见的有standalone ...
- flink on yarn部分源码解析 (FLIP-6 new mode)
我们在https://www.cnblogs.com/dongxiao-yang/p/9403427.html文章里分析了flink提交single job到yarn集群上的代码,flink在1.5版 ...
- Flink的TaskManager启动(源码分析)
通过启动脚本已经找到了TaskManager 的启动类org.apache.flink.runtime.taskexecutor.TaskManagerRunner 来看一下它的main方法中 最后被 ...
- Flink的JobManager启动(源码分析)
都知道Flink中的角色分为Jobmanager,TaskManger 在启动脚本里面已经找到了jobmanager的启动类org.apache.flink.runtime.entrypoint.St ...
- Flink源码阅读(1.7.2)
目录 Client提交任务 flink的图结构 StreamGraph OptimizedPlan JobGraph ExecutionGraph flink部署与执行模型 Single Job Jo ...
- Flink的Job启动TaskManager端(源码分析)
前面说到了 Flink的JobManager启动(源码分析) 启动了TaskManager 然后 Flink的Job启动JobManager端(源码分析) 说到JobManager会将转化得到 ...
- Flink sql 之 TopN 与 StreamPhysicalRankRule (源码解析)
基于flink1.14的源码做解析 公司内有很多业务方都在使用我们Flink sql平台做TopN的计算,今天同事突然问到我,Flink sql 是怎么实现topN的 ? 蒙圈了,这块源码没看过啊 , ...
- [源码分析] 从源码入手看 Flink Watermark 之传播过程
[源码分析] 从源码入手看 Flink Watermark 之传播过程 0x00 摘要 本文将通过源码分析,带领大家熟悉Flink Watermark 之传播过程,顺便也可以对Flink整体逻辑有一个 ...
- Flink源码分析
http://vinoyang.com/ http://wuchong.me Apache Flink源码解析之stream-source https://yq.aliyun.com/articles ...
随机推荐
- ios图层转场动画
动画类型 CAAnimation的子类,用于做转场动画,能够为层提供移出屏幕和移入屏幕的动画效果.iOS比Mac OS X的转场动画效果少一点 UINavigationController就是通过CA ...
- NoSQL生态系统(nosql ecosystem)
Unlike most of the other projects in this book, NoSQL is not a tool, but an ecosystem composed of se ...
- Hadoop相关项目Hive-Pig-Spark-Storm-HBase-Sqoop
Hadoop相关项目Hive-Pig-Spark-Storm-HBase-Sqoop的相关介绍. Hive Pig和Hive的对比 摘要: Pig Pig是一种编程语言,它简化了Hadoop常见的工作 ...
- golang学习笔记 ---TCMalloc
图解 TCMalloc 前言 TCMalloc 是 Google 开发的内存分配器,在不少项目中都有使用,例如在 Golang 中就使用了类似的算法进行内存分配.它具有现代化内存分配器的基本特征:对抗 ...
- MongoDB学习笔记(11) --- 聚合
MongoDB中聚合(aggregate)主要用于处理数据(诸如统计平均值,求和等),并返回计算后的数据结果.有点类似sql语句中的 count(*) aggregate() 方法 MongoDB中聚 ...
- Oracle 12C -- purge dba_recyclebin
SQL> create user abce identified by abce; User created. SQL> grant resource,connect to abce; G ...
- 去掉WinLicense文件效验的方法
去掉WinLicense文件效验的方法 --------------Breakpoints-------------- 地址 模块 活动 反汇编 注释------------------------- ...
- ThinkPHP 3.2 性能优化,实现高性能API开发
需求分析 目前的业务全站使用ThinkPHP 3.2.3,前台.后台.Cli.Api等.目前的业务API访问量数千万,后端7台PHP 5.6,平均CPU使用率20%. 测试数据 真实业务 php5.6 ...
- javascript刷新页面代码
原文链接:http://caibaojian.com/javascript-refresh-page.html reload 方法,该方法强迫浏览器刷新当前页面. 语法:location.reload ...
- 使用C#WebClient类访问(上传/下载/删除/列出文件目录)
在使用WebClient类之前,必须先引用System.Net命名空间,文件下载.上传与删除的都是使用异步编程,也可以使用同步编程, 这里以异步编程为例: 1)文件下载: static void Ma ...