Flink的keyby延时源码
public class RecordWriter<T extends IOReadableWritable> {
==FullBuffer
/**
* This is used to send LatencyMarks to a random target channel.
*/
public void randomEmit(T record) throws IOException, InterruptedException {
sendToTarget(record, rng.nextInt(numChannels));
} private void sendToTarget(T record, int targetChannel) throws IOException, InterruptedException {
RecordSerializer<T> serializer = serializers[targetChannel]; SerializationResult result = serializer.addRecord(record); while (result.isFullBuffer()) {
if (tryFinishCurrentBufferBuilder(targetChannel, serializer)) {
// If this was a full record, we are done. Not breaking
// out of the loop at this point will lead to another
// buffer request before breaking out (that would not be
// a problem per se, but it can lead to stalls in the
// pipeline).
if (result.isFullRecord()) {
break;
}
}
BufferBuilder bufferBuilder = requestNewBufferBuilder(targetChannel); result = serializer.continueWritingWithNextBufferBuilder(bufferBuilder);
}
checkState(!serializer.hasSerializedData(), "All data should be written at once"); if (flushAlways) {
targetPartition.flush(targetChannel);
}
} ==getBufferTimeout
package org.apache.flink.streaming.runtime.tasks;
public abstract class StreamTask<OUT, OP extends StreamOperator<OUT>>
@VisibleForTesting
public static <OUT> List<StreamRecordWriter<SerializationDelegate<StreamRecord<OUT>>>> createStreamRecordWriters(
StreamConfig configuration,
Environment environment) {
List<StreamRecordWriter<SerializationDelegate<StreamRecord<OUT>>>> streamRecordWriters = new ArrayList<>();
List<StreamEdge> outEdgesInOrder = configuration.getOutEdgesInOrder(environment.getUserClassLoader());
Map<Integer, StreamConfig> chainedConfigs = configuration.getTransitiveChainedTaskConfigsWithSelf(environment.getUserClassLoader()); for (int i = 0; i < outEdgesInOrder.size(); i++) {
StreamEdge edge = outEdgesInOrder.get(i);
streamRecordWriters.add(
createStreamRecordWriter(
edge,
i,
environment,
environment.getTaskInfo().getTaskName(),
chainedConfigs.get(edge.getSourceId()).getBufferTimeout()));
}
return streamRecordWriters; } http://vinoyang.com/2016/12/30/flink-runtime-producer-result-partition/
http://vinoyang.com/2017/01/04/flink-runtime-consumer-input-gate/
http://vinoyang.com/2017/01/08/flink-runtime-netty-part-1/
http://vinoyang.com/2017/01/12/flink-runtime-netty-part-2/
http://vinoyang.com/2017/01/15/flink-runtime-netty-part-3/
http://vinoyang.com/2016/12/14/flink-runtime-NetworkEnvironment/
http://vinoyang.com/2016/12/28/flink-runtime-communicate-api/
http://vinoyang.com/archives/2016/12/
http://vinoyang.com/2016/12/20/flink-runtime-unified-data-exchange/
Flink的keyby延时源码的更多相关文章
- flink on yarn部分源码解析
转发请注明原创地址:https://www.cnblogs.com/dongxiao-yang/p/9403427.html flink任务的deploy形式有很多种选择,常见的有standalone ...
- flink on yarn部分源码解析 (FLIP-6 new mode)
我们在https://www.cnblogs.com/dongxiao-yang/p/9403427.html文章里分析了flink提交single job到yarn集群上的代码,flink在1.5版 ...
- Flink的TaskManager启动(源码分析)
通过启动脚本已经找到了TaskManager 的启动类org.apache.flink.runtime.taskexecutor.TaskManagerRunner 来看一下它的main方法中 最后被 ...
- Flink的JobManager启动(源码分析)
都知道Flink中的角色分为Jobmanager,TaskManger 在启动脚本里面已经找到了jobmanager的启动类org.apache.flink.runtime.entrypoint.St ...
- Flink源码阅读(1.7.2)
目录 Client提交任务 flink的图结构 StreamGraph OptimizedPlan JobGraph ExecutionGraph flink部署与执行模型 Single Job Jo ...
- Flink的Job启动TaskManager端(源码分析)
前面说到了 Flink的JobManager启动(源码分析) 启动了TaskManager 然后 Flink的Job启动JobManager端(源码分析) 说到JobManager会将转化得到 ...
- Flink sql 之 TopN 与 StreamPhysicalRankRule (源码解析)
基于flink1.14的源码做解析 公司内有很多业务方都在使用我们Flink sql平台做TopN的计算,今天同事突然问到我,Flink sql 是怎么实现topN的 ? 蒙圈了,这块源码没看过啊 , ...
- [源码分析] 从源码入手看 Flink Watermark 之传播过程
[源码分析] 从源码入手看 Flink Watermark 之传播过程 0x00 摘要 本文将通过源码分析,带领大家熟悉Flink Watermark 之传播过程,顺便也可以对Flink整体逻辑有一个 ...
- Flink源码分析
http://vinoyang.com/ http://wuchong.me Apache Flink源码解析之stream-source https://yq.aliyun.com/articles ...
随机推荐
- 转 安装Nginx 1.2.0+PHP 5.4.3(FastCGI)+MySQL 5.5.24
硬件环境:Dell PowerEdge 1950,4核Intel(R) Xeon(R) CPU E5410 @ 2.33GHz4G内存 系统环境:CentOS release 5.2,内核版本: 2 ...
- 有关python numpy pandas scipy 等 能在YARN集群上 运行PySpark
有关这个问题,似乎这个在某些时候,用python写好,且spark没有响应的算法支持, 能否能在YARN集群上 运行PySpark方式, 将python分析程序提交上去? Spark Applicat ...
- Inno Setup入门(七)——提供安装语言选项
Inno Setup安装目录下有一个Languages的文件夹,该文件夹提供了可供使用的语言,通过在脚本中加入[languages]段,可以实现该项功能,实现代码如下: [setup] ;全局设置,本 ...
- jquery.uploadify 在firefox会出现httperror
原来是因为我的上传处理页面的page 继承了一个基类影响到的 然后这个基类 好像是因为在别的项目里面的原因 希望对也遇到这样的问题的人有帮助咯
- 修改cnblogs日历控件CSS
自定义cnblogs日期控件的样式时, 关于透明度 控件头部死活没法透明 在background属性值后边添加!important即把cnblogs系统自带的属性覆盖, 解决了该问题. !import ...
- Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 2611816 bytes)
一段PHP程序执行报错: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 261181 ...
- 【Java】详解菜单组件
在这篇文章中,笔者会介绍Java图形界面编程中菜单组件的用法.关于菜单组件,因为java存在AWT编程和Swing编程,所以菜单组件也存在AWT菜单和Swing菜单.因为Swing组件使用的比较多,所 ...
- javascript 中 split 函数分割字符串成数组
分割字符串成数组的方法有很多,不过使用最多的还是split函数 <script language="javascript"> str="2,2,3,5,6,6 ...
- python类内init外声明的属性与init内声明的对象属性的访问和操作区别
python类内init外声明的属性与init内声明的对象属性的访问和操作区别(面试题) 1.在ipython中输入以下代码,其输出会是什么? In [1]: class ClassOut: ...: ...
- 常用代码之四:创建jason,jason转换为字符串,字符串转换回jason,c#反序列化jason字符串的几个代码片段
1.创建jason,并JSON.stringify()将之转换为字符串. 直接使用var customer={}, 然后直接customer.属性就可以直接赋值了. 也可以var customer = ...