spark-rpc是如何实现将netty的Channel隐藏在inbox中的
class TransportServer
bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
RpcHandler rpcHandler = appRpcHandler;
for (TransportServerBootstrap bootstrap : bootstraps) {
rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
}
context.initializePipeline(ch, rpcHandler);
}
});
每次有新client连接完成,server都会调用匿名类的initChannel方法,在调用context.initializePipeline(ch, rpcHandler)时,通过createChannelHandler方法,创建TransportResponseHandler用于处理响应消息(Channel直接被隐藏在TransportResponseHandler中),创建TransportRequestHandler 用于处理请求消息(Channel被封装在TransportClient中,TransportClient又被隐藏在TransportRequestHandler),最后返回的TransportChannelHandler包含三者。
public TransportChannelHandler initializePipeline(
SocketChannel channel,
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
channel.pipeline()
.addLast("encoder", encoder)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", decoder)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
// NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
// would require more logic to guarantee if this were not part of the same event loop.
.addLast("handler", channelHandler);
return channelHandler;
} catch (RuntimeException e) {
logger.error("Error while initializing Netty pipeline", e);
throw e;
}
}
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
TransportClient client = new TransportClient(channel, responseHandler);
TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
rpcHandler);
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
}
class TransportChannelHandler
//原始消息到达,根据消息类型调用不同处理方式。此处以RequestMessage消息为例
public void channelRead0(ChannelHandlerContext ctx, Message request) throws Exception {
if (request instanceof RequestMessage) {
requestHandler.handle((RequestMessage) request); //上文实例的TransportRequestHandler
} else {
responseHandler.handle((ResponseMessage) request);
}
}
class TransportRequestHandler
@Override
public void handle(RequestMessage request) {
if (request instanceof ChunkFetchRequest) {
processFetchRequest((ChunkFetchRequest) request);
} else if (request instanceof RpcRequest) {
processRpcRequest((RpcRequest) request); //以rpc请求为例
} else if (request instanceof OneWayMessage) {
processOneWayMessage((OneWayMessage) request);
} else if (request instanceof StreamRequest) {
processStreamRequest((StreamRequest) request);
} else {
throw new IllegalArgumentException("Unknown request type: " + request);
}
}
private void processRpcRequest(final RpcRequest req) {
try {
rpcHandler.receive(reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() { //reverseClient上文传递的值
@Override
public void onSuccess(ByteBuffer response) {
respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
}
@Override
public void onFailure(Throwable e) {
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
}
});
} catch (Exception e) {
logger.error("Error while invoking RpcHandler#receive() on RPC id " + req.requestId, e);
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
} finally {
req.body().release();
}
}
class NettyRpcHandler
override def receive(
client: TransportClient,
message: ByteBuffer,
callback: RpcResponseCallback): Unit = {
val messageToDispatch = internalReceive(client, message) //将client封装在messageToDispatch 中
dispatcher.postRemoteMessage(messageToDispatch, callback) //此方法是将消息封装成inbox形式,放入inbox消息队列
}
internalReceive调用
val requestMessage = nettyEnv.deserialize[RequestMessage](client, message)
class NettyRpcEnv
private[netty] def deserialize[T: ClassTag](client: TransportClient, bytes: ByteBuffer): T = {
NettyRpcEnv.currentClient.withValue(client) {
deserialize { () =>
javaSerializerInstance.deserialize[T](bytes) //反序列化时调用NettyRpcEndpointRef实例的readObject方法,而下面两个DynamicVariable在此处被替换为新值
}
}
}
object NettyRpcEnv
//DynamicVariable可以用来在指定作用域替换数据
private[netty] val currentEnv = new DynamicVariable[NettyRpcEnv](null)
private[netty] val currentClient = new DynamicVariable[TransportClient](null)
class NettyRpcEndpointRef
private def readObject(in: ObjectInputStream): Unit = { //反序列化,为client、nettyEnv填充新值,而非通过网络传送过来的空值
in.defaultReadObject()
nettyEnv = NettyRpcEnv.currentEnv.value
client = NettyRpcEnv.currentClient.value
}
spark-rpc是如何实现将netty的Channel隐藏在inbox中的的更多相关文章
- spark2.1源码分析3:spark-rpc如何实现将netty的Channel隐藏在inbox中
class TransportServer bootstrap.childHandler(new ChannelInitializer<SocketChannel>() { @Overri ...
- Spark RPC框架源码分析(一)简述
Spark RPC系列: Spark RPC框架源码分析(一)运行时序 Spark RPC框架源码分析(二)运行时序 Spark RPC框架源码分析(三)运行时序 一. Spark rpc框架概述 S ...
- Spark RPC框架源码分析(二)RPC运行时序
前情提要: Spark RPC框架源码分析(一)简述 一. Spark RPC概述 上一篇我们已经说明了Spark RPC框架的一个简单例子,Spark RPC相关的两个编程模型,Actor模型和Re ...
- org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse
跑sparkPis示例程序 [root@node01 bin]# ./spark-submit --master spark://node01:7077 --class org.apache.spar ...
- spark RPC详解
前段时间看spark,看着迷迷糊糊的.最近终于有点头绪,先梳理了一下spark rpc相关的东西,先记录下来. 1,概述 个人认为,如果把分布式系统(HDFS, HBASE,SPARK等)比作一个人, ...
- Spark RPC
在Spark中,对于网络调用的底层封装(粘包拆包,编解码,链路管理等)都是在common/network-common包中实现的(详见[common/network-common]).在common/ ...
- spark 源码分析之六--Spark RPC剖析之Dispatcher和Inbox、Outbox剖析
在上篇 spark 源码分析之五 -- Spark内置RPC机制剖析之一创建NettyRPCEnv 中,涉及到了Diapatcher 内容,未做过多的剖析.本篇来剖析一下它的工作原理. Dispatc ...
- spark 源码分析之八--Spark RPC剖析之TransportContext和TransportClientFactory剖析
spark 源码分析之八--Spark RPC剖析之TransportContext和TransportClientFactory剖析 TransportContext 首先官方文档对Transpor ...
- spark 源码分析之十--Spark RPC剖析之TransportResponseHandler、TransportRequestHandler和TransportChannelHandler剖析
spark 源码分析之十--Spark RPC剖析之TransportResponseHandler.TransportRequestHandler和TransportChannelHandler剖析 ...
随机推荐
- MTK平台-抓取蓝牙log
一.MTKLOG抓取 .在拔号键盘输入暗码 *#*##*#* 进入工模EngineerMode .在 Log and Debugging -> MTKLogger 点击开始 .MTKLog存储路 ...
- 安装Feflow遇取的问题
http://www.feflowjs.org/zh-cn/docs/ 在安装 npm install -g feflow-cli 是遇到这要一个问题 i C:\Program Files (x86) ...
- 看书记笔记 书名21天学C#
☆:为重点★:为科普△:注▲:术语 前言概述 ☆一门语言必须包括诸如异常处理,无用单元收集,可扩展数据类型以及代码安全性等特征☆C#特性:简单性,面向对象,模块性,灵活性,简明性 ☆C#面向对象的封装 ...
- Vue常见问题解决办法(一)ESLint检查报错
vue.js报错“Do not use 'new' for side effects“(main.js里)解决办法 ESLint工具检查代码质量,main.js里的原代码是这样的: new Vue({ ...
- fdsafdsafds
name1 = input('请输入一个名字:') name2 = input('请输入一个名字: ') animal = input('请输入一个名字 :') print('刚按完摩的{}出门看见{ ...
- VUE路由携带参数的三种方式
vue 通过路由在进行页面跳转时,会经常携带参数用于同步页面间的数据 路由中携带参数的方式总结如下: 路由定义示例: { name: 'list', path: '/list', component: ...
- 家人的健康和offer的取舍
记得2月份去Amazon面试的时候,小孩子正莫名的发烧,已经破纪录的连续烧了4天,到了6点面试完毕,面试官还试探性的问我还有没有什么要聊的,当时确实是没了心情,就想着回家看小病人,在回家的路上,暗暗的 ...
- 多个字符串有相同的hashcode(没见到大于8的时候转成红黑树)
public static void main(String[] a){ byte[] b1 = {33 , 123 ,124}; byte[] b2 = {33 , 124 , 93}; byte[ ...
- Pac-Man 吃豆人
发售年份 1980 平台 街机 开发商 南梦宫(Namco) 类型 迷宫 https://www.youtube.com/watch?v=dScq4P5gn4A
- 创建一个dynamics 365 CRM online plugin (五) - Images in Plugin
Snapshots of the primary entity's attributes from database before(pre) and after (post) the core pla ...