public class RecordWriter<T extends IOReadableWritable> {

==FullBuffer
/**
* This is used to send LatencyMarks to a random target channel.
*/
public void randomEmit(T record) throws IOException, InterruptedException {
sendToTarget(record, rng.nextInt(numChannels));
} private void sendToTarget(T record, int targetChannel) throws IOException, InterruptedException {
RecordSerializer<T> serializer = serializers[targetChannel]; SerializationResult result = serializer.addRecord(record); while (result.isFullBuffer()) {
if (tryFinishCurrentBufferBuilder(targetChannel, serializer)) {
// If this was a full record, we are done. Not breaking
// out of the loop at this point will lead to another
// buffer request before breaking out (that would not be
// a problem per se, but it can lead to stalls in the
// pipeline).
if (result.isFullRecord()) {
break;
}
}
BufferBuilder bufferBuilder = requestNewBufferBuilder(targetChannel); result = serializer.continueWritingWithNextBufferBuilder(bufferBuilder);
}
checkState(!serializer.hasSerializedData(), "All data should be written at once"); if (flushAlways) {
targetPartition.flush(targetChannel);
}
} ==getBufferTimeout
package org.apache.flink.streaming.runtime.tasks;
public abstract class StreamTask<OUT, OP extends StreamOperator<OUT>>
@VisibleForTesting
public static <OUT> List<StreamRecordWriter<SerializationDelegate<StreamRecord<OUT>>>> createStreamRecordWriters(
StreamConfig configuration,
Environment environment) {
List<StreamRecordWriter<SerializationDelegate<StreamRecord<OUT>>>> streamRecordWriters = new ArrayList<>();
List<StreamEdge> outEdgesInOrder = configuration.getOutEdgesInOrder(environment.getUserClassLoader());
Map<Integer, StreamConfig> chainedConfigs = configuration.getTransitiveChainedTaskConfigsWithSelf(environment.getUserClassLoader()); for (int i = 0; i < outEdgesInOrder.size(); i++) {
StreamEdge edge = outEdgesInOrder.get(i);
streamRecordWriters.add(
createStreamRecordWriter(
edge,
i,
environment,
environment.getTaskInfo().getTaskName(),
chainedConfigs.get(edge.getSourceId()).getBufferTimeout()));
}
return streamRecordWriters; } http://vinoyang.com/2016/12/30/flink-runtime-producer-result-partition/
http://vinoyang.com/2017/01/04/flink-runtime-consumer-input-gate/
http://vinoyang.com/2017/01/08/flink-runtime-netty-part-1/
http://vinoyang.com/2017/01/12/flink-runtime-netty-part-2/
http://vinoyang.com/2017/01/15/flink-runtime-netty-part-3/
http://vinoyang.com/2016/12/14/flink-runtime-NetworkEnvironment/
http://vinoyang.com/2016/12/28/flink-runtime-communicate-api/
http://vinoyang.com/archives/2016/12/
http://vinoyang.com/2016/12/20/flink-runtime-unified-data-exchange/

Flink的keyby延时源码的更多相关文章

  1. flink on yarn部分源码解析

    转发请注明原创地址:https://www.cnblogs.com/dongxiao-yang/p/9403427.html flink任务的deploy形式有很多种选择,常见的有standalone ...

  2. flink on yarn部分源码解析 (FLIP-6 new mode)

    我们在https://www.cnblogs.com/dongxiao-yang/p/9403427.html文章里分析了flink提交single job到yarn集群上的代码,flink在1.5版 ...

  3. Flink的TaskManager启动(源码分析)

    通过启动脚本已经找到了TaskManager 的启动类org.apache.flink.runtime.taskexecutor.TaskManagerRunner 来看一下它的main方法中 最后被 ...

  4. Flink的JobManager启动(源码分析)

    都知道Flink中的角色分为Jobmanager,TaskManger 在启动脚本里面已经找到了jobmanager的启动类org.apache.flink.runtime.entrypoint.St ...

  5. Flink源码阅读(1.7.2)

    目录 Client提交任务 flink的图结构 StreamGraph OptimizedPlan JobGraph ExecutionGraph flink部署与执行模型 Single Job Jo ...

  6. Flink的Job启动TaskManager端(源码分析)

    前面说到了  Flink的JobManager启动(源码分析)  启动了TaskManager 然后  Flink的Job启动JobManager端(源码分析)  说到JobManager会将转化得到 ...

  7. Flink sql 之 TopN 与 StreamPhysicalRankRule (源码解析)

    基于flink1.14的源码做解析 公司内有很多业务方都在使用我们Flink sql平台做TopN的计算,今天同事突然问到我,Flink sql 是怎么实现topN的 ? 蒙圈了,这块源码没看过啊 , ...

  8. [源码分析] 从源码入手看 Flink Watermark 之传播过程

    [源码分析] 从源码入手看 Flink Watermark 之传播过程 0x00 摘要 本文将通过源码分析,带领大家熟悉Flink Watermark 之传播过程,顺便也可以对Flink整体逻辑有一个 ...

  9. Flink源码分析

    http://vinoyang.com/ http://wuchong.me Apache Flink源码解析之stream-source https://yq.aliyun.com/articles ...

随机推荐

  1. 不同类型的磁盘存储在Ubuntu下的性能测试

    Ubuntu下通过lsusb判断USB存储是否是USB3.0: # 要查看Seagate这个移动硬盘 lsusb 或者 lsusb -t $ lsusb Bus Device : ID : Intel ...

  2. poj 2059 单调栈

    题意:求柱状图中最大矩形面积. 单调栈:顾名思义就是栈内元素单调递增的栈. 每次插入数据来维护这个栈,假设当前须要插入的数据小于栈顶的元素,那就一直弹出栈顶的元素.直到满足当前须要插入的元素大于栈顶元 ...

  3. 网站PV、UV以及查看方法

    网站PV.UV以及查看方法 一.名词解释 PV:PV 是Page Views的缩写,即页面浏览量,用户每一次对网站中的每个网页访问均被记录一次.注意,访客每刷新一次页面,pv就增加一次. UV:UV是 ...

  4. 【已解决】mysql连接出错:ERROR 1040 (HY000): Too many connections

    连接mysql,结果出错: ? 1 ERROR 1040 (HY000): Too many connections 去修改mysql的配置文件,然后添加: ? 1 2 3 4 5 6 7 8 9 1 ...

  5. Nginx+FastCGI运行原理(二)

    1.4 PHP与PHP-FPM的安装及优化(2) 标签rlimit_files用于设置PHP-FPM对打开文件描述符的限制,默认值为1024.这个标签的值必须和Linux内核打开文件数关联起来,例如, ...

  6. Using 1-Wire device with Intel Galileo

    Using 1-Wire device with Intel Galileo 3 Replies Many people have had trouble with getting 1-Wire de ...

  7. umdh windbg分析内存泄露

    A.利用工具umdh(user-mode dump heap)分析:此处以程序MemoryLeak.exe为例子 1.开启cmd 键入要定位内存泄露的程序gflags.exe /i memroylea ...

  8. JVM调优总结(一)-- 堆和栈的基本概念

    数据类型 Java虚拟机中,数据类型可以分为两类:基本类型和引用类型.基本类型的变量保存原始值,即:他代表的值就是数值本身:而引用类型的变量保存引用值.“引用值”代表了某个对象的引用,而不是对象本身, ...

  9. 防盗链之URL参数签名 总结

    一.概述 传统的 IP 禁用.referer 防盗链.User-Agent 防盗链.地区访问控制等防盗链措施已经无法完全满足用户要求,所以开发出URL参数签名方式来防盗链 二.实现 Token防盗链是 ...

  10. 【java】详解java中的注解(Annotation)

    目录结构: contents structure [+] 什么是注解 为什么要使用注解 基本语法 4种基本元注解 重复注解 使用注解 运行时处理的注解 编译时处理的注解 1.什么是注解 用一个词就可以 ...