1、An abstract IPC service.  IPC calls take a single {@link Writable} as a parameter, and return a {@link Writable} as their value.  A service runs on a port and is defined by a parameter class and a value class.

2、共有5个内部类ExceptionsHandler,Call,Listener,Responder,Connection,Handler

他需要有一个线程 (Listener)来 accept socket,同时需要一些 Reader线程 来进行 socket 的 read,还有一个 Responder 来进行 socket 的 write,另外,还有若干个 handler线程 来进行业务处理。

3、很多内部属性,位置比较散乱

private final boolean authorize;
private boolean isSecurityEnabled;
private ExceptionsHandler exceptionsHandler = new ExceptionsHandler();

private String bindAddress;
  private int port;                               // port we listen on
  private int handlerCount;                       // number of handler threads
  private int readThreads;                        // number of read threads
  private Class<? extends Writable> paramClass;   // class of call parameters
  private int maxIdleTime;                        // the maximum idle time after  which a client may be disconnected
  private int thresholdIdleConnections;           // the number of idle connections after which we will start cleaning up idle   connections
  int maxConnectionsToNuke;                       // the max number of connections to nuke during a cleanup
  protected RpcInstrumentation rpcMetrics;
  private Configuration conf;
  private SecretManager<TokenIdentifier> secretManager;

private int maxQueueSize;
  private final int maxRespSize;
  private int socketSendBufferSize;
  private final boolean tcpNoDelay; // if T then disable Nagle's Algorithm

volatile private boolean running = true;         // true while server runs
  private BlockingQueue<Call> callQueue; // queued calls

private List<Connection> connectionList =
    Collections.synchronizedList(new LinkedList<Connection>());
  //maintain a list of client connections
  private Listener listener = null;
  private Responder responder = null;
  private int numConnections = 0;
  private Handler[] handlers = null;

4、这个Call和Client里边的Call是不一样的,

A call queued for handling.

private Writable param;                       // the parameter passed
private Connection connection;                // connection to client
private long timestamp;     // the time received when response is null ; the time served when response is not null
private ByteBuffer response;                      // the response for this call

5、Connection

Reads calls from a connection and queues them for handling.

private boolean rpcHeaderRead = false; // if initial rpc header is read
  private boolean headerRead = false;  //if the connection header that follows version is read.

private SocketChannel channel;
  private ByteBuffer data;
  private ByteBuffer dataLengthBuffer;
  private LinkedList<Call> responseQueue;
  private volatile int rpcCount = 0; // number of outstanding rpcs
  private long lastContact;
  private int dataLength;
  private Socket socket;// Cache the remote host & port info so that even if the socket is disconnected, we can say where it used to connect to.
  private String hostAddress;
  private int remotePort;
  private InetAddress addr;
  ConnectionHeader header = new ConnectionHeader();
  Class<?> protocol;
  boolean useSasl;
  SaslServer saslServer;
  private AuthMethod authMethod;
  private boolean saslContextEstablished;
  private boolean skipInitialSaslHandshake;
  private ByteBuffer rpcHeaderBuffer;
  private ByteBuffer unwrappedData;
  private ByteBuffer unwrappedDataLengthBuffer;
 
  UserGroupInformation user = null;
  public UserGroupInformation attemptingUser = null; // user name before auth

// Fake 'call' for failed authorization response
  private final int AUTHROIZATION_FAILED_CALLID = -1;
  private final Call authFailedCall =
    new Call(AUTHROIZATION_FAILED_CALLID, null, this);
  private ByteArrayOutputStream authFailedResponse = new ByteArrayOutputStream();
  // Fake 'call' for SASL context setup
  private static final int SASL_CALLID = -33;
  private final Call saslCall = new Call(SASL_CALLID, null, this);
  private final ByteArrayOutputStream saslResponse = new ByteArrayOutputStream();
 
  private boolean useWrap = false;

6、ExceptionsHandler manages Exception groups for special handling e.g., terse exception group for concise logging messages

7、Handles queued calls .

8、Listens on the socket. Creates jobs for the handler threads

Listener里边有一个内部类,Reader

相应的属性有

private ServerSocketChannel acceptChannel = null; //the accept channel
private Selector selector = null; //the selector that we use for the server
private Reader[] readers = null;
private int currentReader = 0;
private InetSocketAddress address; //the address we bind at
private Random rand = new Random();
private long lastCleanupRunTime = 0; //the last time when a cleanup connec-
                                     //-tion (for idle connections) ran
private long cleanupInterval = 10000; //the minimum interval between
                                      //two cleanup runs
private int backlogLength = conf.getInt("ipc.server.listen.queue.size", 128);
private ExecutorService readPool;

9、Responder

Sends responses of RPC back to clients

相应的属性有

private Selector writeSelector;
    private int pending;         // connections waiting to register
   
    final static int PURGE_INTERVAL = 900000; // 15mins

http://www.cnblogs.com/echomyecho/p/3272238.html

踏着前人的脚印学hadoop——ipc中的Server的更多相关文章

  1. 踏着前人的脚印学hadoop——ipc中的Client

    1.Client有五个内部类,分别是Call,ParallelCall,ParallelResult,Connetion,ConnectionId 其实这五个类就是去完成两件事情的,一件事情是连接,另 ...

  2. 踏着前人的脚印学Hadoop——RPC源码

    A simple RPC mechanism.A protocol  is a Java interface.  All parameters and return types must be one ...

  3. 踏着前人的脚印学Hadoop——结构、重点

    HDFS作为一个分布式文件系统,是所有这些项目的基础.分析好HDFS,有利于了解其他系统.由于Hadoop的HDFS和MapReduce是同一个项目,我们就把他们放在一块,进行分析. 如果把整个had ...

  4. 踏着前人的脚印学Hadoop——序列化,Writerable

    package org.apache.hadoop.io; import java.io.DataOutput;import java.io.DataInput;import java.io.IOEx ...

  5. org.apache.hadoop.ipc.RemoteException(java.io.IOException)

    昨晚突然之间mr跑步起来了 jps查看 进程都在的,但是在reduce任务跑了85%的时候会抛异常 异常情况如下: 2016-09-21 21:32:28,538 INFO [org.apache.h ...

  6. Hadoop IPC的代码结构分析

    与IPC相关的代码在org.apache.hadoop.ipc包下.共七个文件,其中4个辅助类: RemoteException Status VersionedProtocol Connection ...

  7. 一张图解释Hadoop IPC

    基于hadoop2.6.2.... 一张图Server启动,Client访问..... RPC是IPC的一种,IPC还有另外一种LPC,相关请看参考中的3 使用hadoop ipc步骤: 1.定义RP ...

  8. 运行基准测试hadoop集群中的问题:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /benchmarks/TestDFSIO/io_data/test_

    在master(即:host2)中执行 hadoop jar hadoop-test-1.1.2.jar DFSCIOTest -write -nrFiles 12 -fileSize 10240 - ...

  9. hive运行query语句时提示错误:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:

    hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...

随机推荐

  1. kvm相关文章

    配置KVM虚拟机的网络 Bridge和Nat方式http://www.it165.net/os/html/201503/12231.html KVM虚拟机网络配置 Bridge方式,NAT方式 htt ...

  2. C# Socket编程(4)初识Socket和数据流

    经过前面基础知识作为背景,现在对Socket编程进行进一步的学习.在System.Net.Socket命名空间提供了Socket类,利用该类我们可以直接编写Socket的客户端和服务的的程序.但是直接 ...

  3. php 新特性

    PHP 5.6 1.可以使用表达式定义常量 https://php.net/manual/zh/migration56.new-features.php 在之前的 PHP 版本中,必须使用静态值来定义 ...

  4. 微信公共平台php用$GLOBALS["HTTP_RAW_POST_DATA"]收不到信息解决方法

    http://www.cnblogs.com/shendiao/archive/2013/04/24/3041450.html <?php /** * wechat php test */ // ...

  5. C++调用C中编译过的函数要加extern "C"

    C++语言支持函数重载,C语言不支持函数重载.函数被C++编译后在库中的名字与C语言的不同.假设某个C 函数的声明如下:void foo(int x, int y);该函数被C 编译器编译后在库中的名 ...

  6. TWaver HTML5 (2D)--基本概念

    基本概念 TWaver HTML5(以下简称TWaver)使用HTML5技术和javascript语言,可在支持HTML5的浏览器上进行绘图. 使用TWaver前,需熟悉几个基本概念:图元(Eleme ...

  7. Question store (Repeated review)

    题目36 - ACM在线评测系统http://acm.nyist.net/JudgeOnline/problem.php?pid=36 用户名密码INVATION  讲道理太卡 第一:要注意不同的函数 ...

  8. Java线程(二):线程同步synchronized和volatile

    上篇通过一个简单的例子说明了线程安全与不安全,在例子中不安全的情况下输出的结果恰好是逐个递增的(其实是巧合,多运行几次,会产生不同的输出结果),为什么会产生这样的结果呢,因为建立的Count对象是线程 ...

  9. Qt之QThread(深入理解)

    简述 为了让程序尽快响应用户操作,在开发应用程序时经常会使用到线程.对于耗时操作如果不使用线程,UI界面将会长时间处于停滞状态,这种情况是用户非常不愿意看到的,我们可以用线程来解决这个问题. 前面,已 ...

  10. javaSE之如何将一个文件复制到另一个文件

    /* * (1). 文件字符输入,输出流 * 文件字节输入,输出流的read和write方法使用 * 字节数组读写数据,即以字节为单位处理数据,因此,字节流不能很好的操作Unicode字符 * ,比如 ...