踏着前人的脚印学hadoop——ipc中的Server
1、An abstract IPC service. IPC calls take a single {@link Writable} as a parameter, and return a {@link Writable} as their value. A service runs on a port and is defined by a parameter class and a value class.
2、共有5个内部类ExceptionsHandler,Call,Listener,Responder,Connection,Handler
他需要有一个线程 (Listener)来 accept socket,同时需要一些 Reader线程 来进行 socket 的 read,还有一个 Responder 来进行 socket 的 write,另外,还有若干个 handler线程 来进行业务处理。
3、很多内部属性,位置比较散乱
private final boolean authorize;
private boolean isSecurityEnabled;
private ExceptionsHandler exceptionsHandler = new ExceptionsHandler();
private String bindAddress;
private int port; // port we listen on
private int handlerCount; // number of handler threads
private int readThreads; // number of read threads
private Class<? extends Writable> paramClass; // class of call parameters
private int maxIdleTime; // the maximum idle time after which a client may be disconnected
private int thresholdIdleConnections; // the number of idle connections after which we will start cleaning up idle connections
int maxConnectionsToNuke; // the max number of connections to nuke during a cleanup
protected RpcInstrumentation rpcMetrics;
private Configuration conf;
private SecretManager<TokenIdentifier> secretManager;
private int maxQueueSize;
private final int maxRespSize;
private int socketSendBufferSize;
private final boolean tcpNoDelay; // if T then disable Nagle's Algorithm
volatile private boolean running = true; // true while server runs
private BlockingQueue<Call> callQueue; // queued calls
private List<Connection> connectionList =
Collections.synchronizedList(new LinkedList<Connection>());
//maintain a list of client connections
private Listener listener = null;
private Responder responder = null;
private int numConnections = 0;
private Handler[] handlers = null;
4、这个Call和Client里边的Call是不一样的,
A call queued for handling.
private Writable param; // the parameter passed
private Connection connection; // connection to client
private long timestamp; // the time received when response is null ; the time served when response is not null
private ByteBuffer response; // the response for this call
5、Connection
Reads calls from a connection and queues them for handling.
private boolean rpcHeaderRead = false; // if initial rpc header is read
private boolean headerRead = false; //if the connection header that follows version is read.
private SocketChannel channel;
private ByteBuffer data;
private ByteBuffer dataLengthBuffer;
private LinkedList<Call> responseQueue;
private volatile int rpcCount = 0; // number of outstanding rpcs
private long lastContact;
private int dataLength;
private Socket socket;// Cache the remote host & port info so that even if the socket is disconnected, we can say where it used to connect to.
private String hostAddress;
private int remotePort;
private InetAddress addr;
ConnectionHeader header = new ConnectionHeader();
Class<?> protocol;
boolean useSasl;
SaslServer saslServer;
private AuthMethod authMethod;
private boolean saslContextEstablished;
private boolean skipInitialSaslHandshake;
private ByteBuffer rpcHeaderBuffer;
private ByteBuffer unwrappedData;
private ByteBuffer unwrappedDataLengthBuffer;
UserGroupInformation user = null;
public UserGroupInformation attemptingUser = null; // user name before auth
// Fake 'call' for failed authorization response
private final int AUTHROIZATION_FAILED_CALLID = -1;
private final Call authFailedCall =
new Call(AUTHROIZATION_FAILED_CALLID, null, this);
private ByteArrayOutputStream authFailedResponse = new ByteArrayOutputStream();
// Fake 'call' for SASL context setup
private static final int SASL_CALLID = -33;
private final Call saslCall = new Call(SASL_CALLID, null, this);
private final ByteArrayOutputStream saslResponse = new ByteArrayOutputStream();
private boolean useWrap = false;
6、ExceptionsHandler manages Exception groups for special handling e.g., terse exception group for concise logging messages
7、Handles queued calls .
8、Listens on the socket. Creates jobs for the handler threads
Listener里边有一个内部类,Reader
相应的属性有
private ServerSocketChannel acceptChannel = null; //the accept channel
private Selector selector = null; //the selector that we use for the server
private Reader[] readers = null;
private int currentReader = 0;
private InetSocketAddress address; //the address we bind at
private Random rand = new Random();
private long lastCleanupRunTime = 0; //the last time when a cleanup connec-
//-tion (for idle connections) ran
private long cleanupInterval = 10000; //the minimum interval between
//two cleanup runs
private int backlogLength = conf.getInt("ipc.server.listen.queue.size", 128);
private ExecutorService readPool;
9、Responder
Sends responses of RPC back to clients
相应的属性有
private Selector writeSelector;
private int pending; // connections waiting to register
final static int PURGE_INTERVAL = 900000; // 15mins
http://www.cnblogs.com/echomyecho/p/3272238.html
踏着前人的脚印学hadoop——ipc中的Server的更多相关文章
- 踏着前人的脚印学hadoop——ipc中的Client
1.Client有五个内部类,分别是Call,ParallelCall,ParallelResult,Connetion,ConnectionId 其实这五个类就是去完成两件事情的,一件事情是连接,另 ...
- 踏着前人的脚印学Hadoop——RPC源码
A simple RPC mechanism.A protocol is a Java interface. All parameters and return types must be one ...
- 踏着前人的脚印学Hadoop——结构、重点
HDFS作为一个分布式文件系统,是所有这些项目的基础.分析好HDFS,有利于了解其他系统.由于Hadoop的HDFS和MapReduce是同一个项目,我们就把他们放在一块,进行分析. 如果把整个had ...
- 踏着前人的脚印学Hadoop——序列化,Writerable
package org.apache.hadoop.io; import java.io.DataOutput;import java.io.DataInput;import java.io.IOEx ...
- org.apache.hadoop.ipc.RemoteException(java.io.IOException)
昨晚突然之间mr跑步起来了 jps查看 进程都在的,但是在reduce任务跑了85%的时候会抛异常 异常情况如下: 2016-09-21 21:32:28,538 INFO [org.apache.h ...
- Hadoop IPC的代码结构分析
与IPC相关的代码在org.apache.hadoop.ipc包下.共七个文件,其中4个辅助类: RemoteException Status VersionedProtocol Connection ...
- 一张图解释Hadoop IPC
基于hadoop2.6.2.... 一张图Server启动,Client访问..... RPC是IPC的一种,IPC还有另外一种LPC,相关请看参考中的3 使用hadoop ipc步骤: 1.定义RP ...
- 运行基准测试hadoop集群中的问题:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /benchmarks/TestDFSIO/io_data/test_
在master(即:host2)中执行 hadoop jar hadoop-test-1.1.2.jar DFSCIOTest -write -nrFiles 12 -fileSize 10240 - ...
- hive运行query语句时提示错误:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:
hive> select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1 Launching Jo ...
随机推荐
- kvm相关文章
配置KVM虚拟机的网络 Bridge和Nat方式http://www.it165.net/os/html/201503/12231.html KVM虚拟机网络配置 Bridge方式,NAT方式 htt ...
- C# Socket编程(4)初识Socket和数据流
经过前面基础知识作为背景,现在对Socket编程进行进一步的学习.在System.Net.Socket命名空间提供了Socket类,利用该类我们可以直接编写Socket的客户端和服务的的程序.但是直接 ...
- php 新特性
PHP 5.6 1.可以使用表达式定义常量 https://php.net/manual/zh/migration56.new-features.php 在之前的 PHP 版本中,必须使用静态值来定义 ...
- 微信公共平台php用$GLOBALS["HTTP_RAW_POST_DATA"]收不到信息解决方法
http://www.cnblogs.com/shendiao/archive/2013/04/24/3041450.html <?php /** * wechat php test */ // ...
- C++调用C中编译过的函数要加extern "C"
C++语言支持函数重载,C语言不支持函数重载.函数被C++编译后在库中的名字与C语言的不同.假设某个C 函数的声明如下:void foo(int x, int y);该函数被C 编译器编译后在库中的名 ...
- TWaver HTML5 (2D)--基本概念
基本概念 TWaver HTML5(以下简称TWaver)使用HTML5技术和javascript语言,可在支持HTML5的浏览器上进行绘图. 使用TWaver前,需熟悉几个基本概念:图元(Eleme ...
- Question store (Repeated review)
题目36 - ACM在线评测系统http://acm.nyist.net/JudgeOnline/problem.php?pid=36 用户名密码INVATION 讲道理太卡 第一:要注意不同的函数 ...
- Java线程(二):线程同步synchronized和volatile
上篇通过一个简单的例子说明了线程安全与不安全,在例子中不安全的情况下输出的结果恰好是逐个递增的(其实是巧合,多运行几次,会产生不同的输出结果),为什么会产生这样的结果呢,因为建立的Count对象是线程 ...
- Qt之QThread(深入理解)
简述 为了让程序尽快响应用户操作,在开发应用程序时经常会使用到线程.对于耗时操作如果不使用线程,UI界面将会长时间处于停滞状态,这种情况是用户非常不愿意看到的,我们可以用线程来解决这个问题. 前面,已 ...
- javaSE之如何将一个文件复制到另一个文件
/* * (1). 文件字符输入,输出流 * 文件字节输入,输出流的read和write方法使用 * 字节数组读写数据,即以字节为单位处理数据,因此,字节流不能很好的操作Unicode字符 * ,比如 ...