class  TransportServer

 bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
RpcHandler rpcHandler = appRpcHandler;
for (TransportServerBootstrap bootstrap : bootstraps) {
rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
}
context.initializePipeline(ch, rpcHandler);
}
});

每次有新client连接完成,server都会调用匿名类的initChannel方法,在调用context.initializePipeline(ch, rpcHandler)时,通过createChannelHandler方法,创建TransportResponseHandler用于处理响应消息(Channel直接被隐藏在TransportResponseHandler中),创建TransportRequestHandler 用于处理请求消息(Channel被封装在TransportClient中,TransportClient又被隐藏在TransportRequestHandler),最后返回的TransportChannelHandler包含三者。

public TransportChannelHandler initializePipeline(
SocketChannel channel,
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
channel.pipeline()
.addLast("encoder", encoder)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", decoder)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
// NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
// would require more logic to guarantee if this were not part of the same event loop.
.addLast("handler", channelHandler);
return channelHandler;
} catch (RuntimeException e) {
logger.error("Error while initializing Netty pipeline", e);
throw e;
}
}
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
TransportClient client = new TransportClient(channel, responseHandler);
TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
rpcHandler);
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
}
class TransportChannelHandler
//原始消息到达,根据消息类型调用不同处理方式。此处以RequestMessage消息为例
public void channelRead0(ChannelHandlerContext ctx, Message request) throws Exception {
if (request instanceof RequestMessage) {
requestHandler.handle((RequestMessage) request); //上文实例的TransportRequestHandler
} else {
responseHandler.handle((ResponseMessage) request);
}
}
class  TransportRequestHandler 

  @Override
public void handle(RequestMessage request) {
if (request instanceof ChunkFetchRequest) {
processFetchRequest((ChunkFetchRequest) request);
} else if (request instanceof RpcRequest) {
processRpcRequest((RpcRequest) request); //以rpc请求为例
} else if (request instanceof OneWayMessage) {
processOneWayMessage((OneWayMessage) request);
} else if (request instanceof StreamRequest) {
processStreamRequest((StreamRequest) request);
} else {
throw new IllegalArgumentException("Unknown request type: " + request);
}
} private void processRpcRequest(final RpcRequest req) {
try {
rpcHandler.receive(reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() { //reverseClient上文传递的值
@Override
public void onSuccess(ByteBuffer response) {
respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
} @Override
public void onFailure(Throwable e) {
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
}
});
} catch (Exception e) {
logger.error("Error while invoking RpcHandler#receive() on RPC id " + req.requestId, e);
respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
} finally {
req.body().release();
}
} class NettyRpcHandler override def receive(
client: TransportClient,
message: ByteBuffer,
callback: RpcResponseCallback): Unit = {
val messageToDispatch = internalReceive(client, message) //将client封装在messageToDispatch 中
dispatcher.postRemoteMessage(messageToDispatch, callback) //此方法是将消息封装成inbox形式,放入inbox消息队列
} internalReceive调用
val requestMessage = nettyEnv.deserialize[RequestMessage](client, message) class NettyRpcEnv
private[netty] def deserialize[T: ClassTag](client: TransportClient, bytes: ByteBuffer): T = {
NettyRpcEnv.currentClient.withValue(client) {
deserialize { () =>
javaSerializerInstance.deserialize[T](bytes) //反序列化时调用NettyRpcEndpointRef实例的readObject方法,而下面两个DynamicVariable在此处被替换为新值
}
}
} object NettyRpcEnv
//DynamicVariable可以用来在指定作用域替换数据
private[netty] val currentEnv = new DynamicVariable[NettyRpcEnv](null)
private[netty] val currentClient = new DynamicVariable[TransportClient](null) class NettyRpcEndpointRef private def readObject(in: ObjectInputStream): Unit = { //反序列化,为client、nettyEnv填充新值,而非通过网络传送过来的空值
in.defaultReadObject()
nettyEnv = NettyRpcEnv.currentEnv.value
client = NettyRpcEnv.currentClient.value
}

spark-rpc是如何实现将netty的Channel隐藏在inbox中的的更多相关文章

  1. spark2.1源码分析3:spark-rpc如何实现将netty的Channel隐藏在inbox中

    class TransportServer bootstrap.childHandler(new ChannelInitializer<SocketChannel>() { @Overri ...

  2. Spark RPC框架源码分析(一)简述

    Spark RPC系列: Spark RPC框架源码分析(一)运行时序 Spark RPC框架源码分析(二)运行时序 Spark RPC框架源码分析(三)运行时序 一. Spark rpc框架概述 S ...

  3. Spark RPC框架源码分析(二)RPC运行时序

    前情提要: Spark RPC框架源码分析(一)简述 一. Spark RPC概述 上一篇我们已经说明了Spark RPC框架的一个简单例子,Spark RPC相关的两个编程模型,Actor模型和Re ...

  4. org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse

    跑sparkPis示例程序 [root@node01 bin]# ./spark-submit --master spark://node01:7077 --class org.apache.spar ...

  5. spark RPC详解

    前段时间看spark,看着迷迷糊糊的.最近终于有点头绪,先梳理了一下spark rpc相关的东西,先记录下来. 1,概述 个人认为,如果把分布式系统(HDFS, HBASE,SPARK等)比作一个人, ...

  6. Spark RPC

    在Spark中,对于网络调用的底层封装(粘包拆包,编解码,链路管理等)都是在common/network-common包中实现的(详见[common/network-common]).在common/ ...

  7. spark 源码分析之六--Spark RPC剖析之Dispatcher和Inbox、Outbox剖析

    在上篇 spark 源码分析之五 -- Spark内置RPC机制剖析之一创建NettyRPCEnv 中,涉及到了Diapatcher 内容,未做过多的剖析.本篇来剖析一下它的工作原理. Dispatc ...

  8. spark 源码分析之八--Spark RPC剖析之TransportContext和TransportClientFactory剖析

    spark 源码分析之八--Spark RPC剖析之TransportContext和TransportClientFactory剖析 TransportContext 首先官方文档对Transpor ...

  9. spark 源码分析之十--Spark RPC剖析之TransportResponseHandler、TransportRequestHandler和TransportChannelHandler剖析

    spark 源码分析之十--Spark RPC剖析之TransportResponseHandler.TransportRequestHandler和TransportChannelHandler剖析 ...

随机推荐

  1. sublime text常用快捷键及多行光标批量操作教程

    sublime text常用快捷键及多行光标批量操作教程   版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/a772304419/article/d ...

  2. html基础js

    HTML中的三把利器的JS 又称为JavaScript,看着好像和Java有点联系,实际上他和java半毛钱关系都没有,JavaScript和我们学习的Python.Go.Java.C++等,都是一种 ...

  3. eclipse中js报错简单快捷的解决方式

    eclipse中对正确的js文件报错十分常见,我的项目中只要是以.js结尾的必会报错,作为一名小小的程序员,看到“满江红”甚是烦躁!今天就给大家分享一个方便又快捷的解决方案. 瞄准被报错的js文件点鼠 ...

  4. myEclipse出现cannot paste the clipboard contents into the selected elements报错

    导入jar包报错,cannot paste the clipboard contents into the selected elements,查阅资料让重新打开工程,但依然报错. 最后在本地路径复制 ...

  5. centos安装实用总结

    1.常用软件安装: yum install -y bash-completion vim lrzsz wget expect net-tools nc nmap tree dos2unix htop ...

  6. 使用数据泵expdp、impdp备份和还原oracle数据库

    前面我已经整理过EXP 和 IMP备份和还原Oracle数据库的方法 今天我们只讲使用数据泵 expdp 和impdp的方法,有的同学会问他们有什么差别呢? EXP和IMP是客户端工具程序,它们既可以 ...

  7. cisco4507引擎模式切换

    1.redu     mode sso2.wri 可能存在的问题:无法切换至sso原因:ios镜像版本不一致 解决方法: 1. copy bootflash: slavebootflash: 2. d ...

  8. sklearn learn preprocessing

    train_test_split sklearn.model_selection.train_test_split(*arrays, test_size(float,int/None),#defaul ...

  9. java基本数据类型和运算符

    一.基本数据类型 种类: 内置数据类型 引用数据类型 1.内置数据类型 一共有八种基本类型,六个数字类型(四个整数类型,两个浮点型),一个布尔型,一个字符类型. (1)byte: byte数据类型是8 ...

  10. 【java编程】java的关键字修饰符

    一.transient java语言的关键字,变量修饰符,如果用transient声明一个实例变量,当对象存储时,它的值不需要维持.换句话来说就是,用transient关键字标记的成员变量不参与序列化 ...