spark源码阅读之network（2）

在上节的解读中发现spark的源码中大量使用netty的buffer部分的api,该节将看到netty核心的一些api,比如channel:

在Netty里，Channel是通讯的载体(网络套接字或组件的连接)，而ChannelHandler负责Channel中的逻辑处理,channel支持读，写，绑定本地端口，连接远程等，Channel中所有的操作都是异步的，当发生io操作的时候将会返回一个ChannelFutrue的接口，在ChannelFutrue里面可以处理操作成功、失败、取消后的动作。有了这些理解就可以看client部分的源码了

 

TransportClientFactory是一个创建TransportClient的工厂类，该类为每一个网络地址都提供了一个连接池，相同的主机返回相同的TransportClient。所有的TransportClient都共享一个EventLoopGroup,该类用于处理channel的上的事件的。

1. privatestaticclassClientPool{
2. TransportClient[] clients;
3. Object[] locks;
5. publicClientPool(int size){
6. clients =newTransportClient[size];
7. locks =newObject[size];
8. for(int i =0; i < size; i++){
9. locks[i]=newObject();
10. }
11. }

ClientPool表示一个连接池，每一个地址对应一个连接池。连接池的大小spark.shuffle.io.numConnectionsPerPeer来指定。该连接池怎么使用呢。用户传递一个地址进来，该地址作为key到connectionPool中查找该地址对应的连接池，没有就生成一个，获取连接池后需要随机的获取一个连接，这个时候连接池中锁就用到了。

1. publicTransportClient createClient(String remoteHost,int remotePort)throwsIOException{
2. // Get connection from the connection pool first.
3. // If it is not found or not active, create a new one.
4. finalInetSocketAddress address =newInetSocketAddress(remoteHost, remotePort);
6. // Create the ClientPool if we don't have it yet.
7. ClientPool clientPool = connectionPool.get(address);
8. if(clientPool ==null){
9. connectionPool.putIfAbsent(address,newClientPool(numConnectionsPerPeer));
10. clientPool = connectionPool.get(address);
11. }
13. int clientIndex = rand.nextInt(numConnectionsPerPeer);
14. TransportClient cachedClient = clientPool.clients[clientIndex];
16. if(cachedClient !=null&& cachedClient.isActive()){
17. logger.trace("Returning cached connection to {}: {}", address, cachedClient);
18. return cachedClient;
19. }
21. // If we reach here, we don't have an existing connection open. Let's create a new one.
22. // Multiple threads might race here to create new connections. Keep only one of them active.
23. synchronized(clientPool.locks[clientIndex]){
24. cachedClient = clientPool.clients[clientIndex];
26. if(cachedClient !=null){
27. if(cachedClient.isActive()){
28. logger.trace("Returning cached connection to {}: {}", address, cachedClient);
29. return cachedClient;
30. }else{
31. logger.info("Found inactive connection to {}, creating a new one.", address);
32. }
33. }
34. clientPool.clients[clientIndex]= createClient(address);
35. return clientPool.clients[clientIndex];
36. }
37. }

怎么创建一个全新的TransportClient，这块要使用netty的bootstrap类帮忙了，主要bootstrap的配置，缓存分配器使用缓存池来管理。调用bootstrap的handler函数给bootstrap添加了一个ChannelHandler，当bootstrap连接成功后回调该ChannelHandler，在ChannelHandler的initchannel的监听方法里面获取了连接通道SocketChannel，使用TransportContext的initializePipeline来初始化通道，就是给通道添加监听器。在这个方法里面我们拿到了

TransportClient和channel。

为什么要使用AtomicReference来保存他们的引用呢？

说说我的理解：内部类只能使用外部类的final变量，局部final变量必须声明的时候初始化，如果不使用AtomicReference就无法保持内部类的一些对象的引用。

1. */
2. publicTransportClient createUnmanagedClient(String remoteHost,int remotePort)
3. throwsIOException{
4. finalInetSocketAddress address =newInetSocketAddress(remoteHost, remotePort);
5. return createClient(address);
6. }
8. /** Create a completely new {@link TransportClient} to the remote address. */
9. privateTransportClient createClient(InetSocketAddress address)throwsIOException{
10. logger.debug("Creating new connection to "+ address);
12. Bootstrap bootstrap =newBootstrap();
13. bootstrap.group(workerGroup)
14. .channel(socketChannelClass)
15. // Disable Nagle's Algorithm since we don't want packets to wait
16. .option(ChannelOption.TCP_NODELAY,true)
17. .option(ChannelOption.SO_KEEPALIVE,true)
18. .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, conf.connectionTimeoutMs())
19. .option(ChannelOption.ALLOCATOR, pooledAllocator);
21. finalAtomicReference<TransportClient> clientRef =newAtomicReference<TransportClient>();
22. finalAtomicReference<Channel> channelRef =newAtomicReference<Channel>();
24. bootstrap.handler(newChannelInitializer<SocketChannel>(){
25. @Override
26. publicvoid initChannel(SocketChannel ch){
27. TransportChannelHandler clientHandler = context.initializePipeline(ch);
28. clientRef.set(clientHandler.getClient());
29. channelRef.set(ch);
30. }
31. });
33. // Connect to the remote server
34. long preConnect =System.nanoTime();
35. ChannelFuture cf = bootstrap.connect(address);
36. if(!cf.awaitUninterruptibly(conf.connectionTimeoutMs())){
37. thrownewIOException(
38. String.format("Connecting to %s timed out (%s ms)", address, conf.connectionTimeoutMs()));
39. }elseif(cf.cause()!=null){
40. thrownewIOException(String.format("Failed to connect to %s", address), cf.cause());
41. }
43. TransportClient client = clientRef.get();
44. Channel channel = channelRef.get();
45. assert client !=null:"Channel future completed successfully with null client";
47. // Execute any client bootstraps synchronously before marking the Client as successful.
48. long preBootstrap =System.nanoTime();
49. logger.debug("Connection to {} successful, running bootstraps...", address);
50. try{
51. for(TransportClientBootstrap clientBootstrap : clientBootstraps){
52. clientBootstrap.doBootstrap(client, channel);
53. }
54. }catch(Exception e){// catch non-RuntimeExceptions too as bootstrap may be written in Scala
55. long bootstrapTimeMs =(System.nanoTime()- preBootstrap)/1000000;
56. logger.error("Exception while bootstrapping client after "+ bootstrapTimeMs +" ms", e);
57. client.close();
58. throwThrowables.propagate(e);
59. }
60. long postBootstrap =System.nanoTime();
62. logger.debug("Successfully created connection to {} after {} ms ({} ms spent in bootstraps)",
63. address,(postBootstrap - preConnect)/1000000,(postBootstrap - preBootstrap)/1000000);
65. return client;
66. }

 

下面看下TransportClient，该类有两个作用：获取数据和发送请求，获取数据用来获取预先协议好的数据流，把数据打散成块（大小为KB和MB)便于传输，当TransportClient要从流上获取数据时，流相关的配置不是TCP的传输层做的，而是需要调用TransportClient的sendRPC执行一些配置。具体流程如下：

client.sendRPC(new OpenFile("/foo") 返回一个StreamId = 10

client.fetchChunk(streamId=100,chunkIndex= 0,callback)

client.fetchChunk(streamId=100,chunkIndex= 1,callback)

client.sendRPC(new CloseStream(100))

一个TransportClient可以使用在多个Streams上，但是一个streams只能和一个client绑定，以免响应顺序错乱。

一个client有3个成员变量：channel用于写操作，向服务器端发送请求，TransportResponseHandler用于处理服务器端响应，clientId给client编号。

1. privatefinalChannel channel;
2. privatefinalTransportResponseHandler handler;
3. @NullableprivateString clientId;

client有3个请求函数，一个是请求数据流中的一个数据块，用于数据传输，第二个是请求整个数据流，用于数据传输，第三个是发送控制请求。有点像ftp,一个用于控制，一个用于数据。

1. publicvoid fetchChunk(
2. long streamId,
3. finalint chunkIndex,
4. finalChunkReceivedCallback callback){
5. finalString serverAddr =NettyUtils.getRemoteAddress(channel);
6. finallong startTime =System.currentTimeMillis();
7. logger.debug("Sending fetch chunk request {} to {}", chunkIndex, serverAddr);
9. finalStreamChunkId streamChunkId =newStreamChunkId(streamId, chunkIndex);
10. handler.addFetchRequest(streamChunkId, callback);
12. channel.writeAndFlush(newChunkFetchRequest(streamChunkId)).addListener(
13. newChannelFutureListener(){
14. @Override
15. publicvoid operationComplete(ChannelFuture future)throwsException{
16. if(future.isSuccess()){
17. long timeTaken =System.currentTimeMillis()- startTime;
18. logger.trace("Sending request {} to {} took {} ms", streamChunkId, serverAddr,
19. timeTaken);
20. }else{
21. String errorMsg =String.format("Failed to send request %s to %s: %s", streamChunkId,
22. serverAddr, future.cause());
23. logger.error(errorMsg, future.cause());
24. handler.removeFetchRequest(streamChunkId);
25. channel.close();
26. try{
27. callback.onFailure(chunkIndex,newIOException(errorMsg, future.cause()));
28. }catch(Exception e){
29. logger.error("Uncaught exception in RPC response callback handler!", e);
30. }
31. }
32. }
33. });
34. }

callback有两个方法，这里要说明下他们的回调机制，onFailure在channel的IO操作失败后调用，就是ChannelFuture失败时候调用，ChannelFuture是IO操作的结果。

onSuccess调用时在channel的事件处理流程中使用，context.initializePipeline(ch)给channel注册了一个TransportChannelHandler，TransportChannelHandler包含了TransportResponseHandler对象，它把响应结果转发给TransportResponseHandler用于处理服务器端响应，handler.addFetchRequest(streamChunkId, callback)映射每个响应对应的回调接口。在对应响应到来时调用对应回调接口。

channel.writeAndFlush的对象需要实现Encodable接口。该接口的一些方法被MessageDecoder和MessageEncoder调用

 

sendRpc和上面方法一样，这里就不描述了，看下stream方法

1. publicvoid stream(finalString streamId,finalStreamCallback callback){
2. finalString serverAddr =NettyUtils.getRemoteAddress(channel);
3. finallong startTime =System.currentTimeMillis();
4. logger.debug("Sending stream request for {} to {}", streamId, serverAddr);
6. // Need to synchronize here so that the callback is added to the queue and the RPC is
7. // written to the socket atomically, so that callbacks are called in the right order
8. // when responses arrive.
9. synchronized(this){
10. handler.addStreamCallback(callback);
11. channel.writeAndFlush(newStreamRequest(streamId)).addListener(
12. newChannelFutureListener(){
13. @Override
14. publicvoid operationComplete(ChannelFuture future)throwsException{
15. if(future.isSuccess()){
16. long timeTaken =System.currentTimeMillis()- startTime;
17. logger.trace("Sending request for {} to {} took {} ms", streamId, serverAddr,
18. timeTaken);
19. }else{
20. String errorMsg =String.format("Failed to send request for %s to %s: %s", streamId,
21. serverAddr, future.cause());
22. logger.error(errorMsg, future.cause());
23. channel.close();
24. try{
25. callback.onFailure(streamId,newIOException(errorMsg, future.cause()));
26. }catch(Exception e){
27. logger.error("Uncaught exception in RPC response callback handler!", e);
28. }
29. }
30. }
31. });
32. }
33. }

这里加了一个同步块，这样保证回调接口调用和请求的顺序一样。这里一个不明白的地方就是，同时发两个请求，第二个请求可能比第一个请求更快返回。怎么保证顺序一致呢？

 

sendRpcSysnc是一个非常有意思的方法，这里学习了Future怎么使用了。

1. publicbyte[] sendRpcSync(byte[] message,long timeoutMs){
2. finalSettableFuture<byte[]> result =SettableFuture.create();
4. sendRpc(message,newRpcResponseCallback(){
5. @Override
6. publicvoid onSuccess(byte[] response){
7. result.set(response);
8. }
10. @Override
11. publicvoid onFailure(Throwable e){
12. result.setException(e);
13. }
14. });
16. try{
17. return result.get(timeoutMs,TimeUnit.MILLISECONDS);
18. }catch(ExecutionException e){
19. throwThrowables.propagate(e.getCause());
20. }catch(Exception e){
21. throwThrowables.propagate(e);
22. }
23. }

匿名内部类智能使用外部类的final,要异步获取内部类的数据使用了一个SettableFuture。

 

 

 

 

 

 

 

 

 

 

 

 

 

来自为知笔记(Wiz)