无意间多瞄了一眼hbase0.98的代码,想复习下put流程。发现htable里面已经找不到processBatchOfPuts()奇怪了。看了半天原来变化还真大事实上0.96就没这个了,于是又搞了个0.96的代码看看

之前有篇能够对照差异,请转移至:http://blog.csdn.net/luyee2010/article/details/8435739 只是排版太乱将就看吧。

HTable.java

  public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
//批量
@Override
public void put(final List<Put> puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}

这里writeAsyncBuffer已经替换了原来的 writeBuffer,事实上仅仅是名字不同

  private void doPut(Put put) throws InterruptedIOException, RetriesExhaustedWithDetailsException {
if (ap.hasError()){
backgroundFlushCommits(true);
} validatePut(put); currentWriteBufferSize += put.heapSize();
writeAsyncBuffer.add(put); while (currentWriteBufferSize > writeBufferSize) {
backgroundFlushCommits(false);
}
}

这里应该是backgroundFlushCommits与原来的flushCommits()差点儿相同,but跟踪进去,卧槽!,都是哪跟哪了,差异有点大。之前一行来着

connection.processBatchOfPuts(writeBuffer, tableName, pool);

1。假设当前writeAsyncBuffer不为空或者之前没运行无错误,提交writeAsyncBuffer

  private void backgroundFlushCommits(boolean synchronous) throws
InterruptedIOException, RetriesExhaustedWithDetailsException { try {
// If there is an error on the operations in progress, we don't add new operations.
if (writeAsyncBuffer.size() > 0 && !ap.hasError()) {
ap.submit(writeAsyncBuffer, true);
} if (synchronous || ap.hasError()) {
if (ap.hasError() && LOG.isDebugEnabled()) {
LOG.debug(tableName + ": One or more of the operations have failed -" +
" waiting for all operation in progress to finish (successfully or not)");
}
ap.waitUntilDone();
} if (ap.hasError()) {
if (!clearBufferOnFail) {
// if clearBufferOnFailed is not set, we're supposed to keep the failed operation in the
// write buffer. This is a questionable feature kept here for backward compatibility
writeAsyncBuffer.addAll(ap.getFailedOperations());
}
RetriesExhaustedWithDetailsException e = ap.getErrors();
ap.clearErrors();
throw e;
}
} finally {
currentWriteBufferSize = 0;
for (Row mut : writeAsyncBuffer) {
if (mut instanceof Mutation) {
currentWriteBufferSize += ((Mutation) mut).heapSize();
}
}
}
}

这个backgroundFlushCommits看了好久也没看出个啥来。仅仅好跟ap.submit(writeAsyncBuffer, true);

看到这个

Map<HRegionLocation, MultiAction<Row>> actionsByServer =new HashMap<HRegionLocation, MultiAction<Row>>();

的时候感觉一下子有希望了,这个应该跟之前的像吧!

  public void submit(List<? extends Row> rows, boolean atLeastOne) throws InterruptedIOException {
if (rows.isEmpty()) {
return;
} // This looks like we are keying by region but HRegionLocation has a comparator that compares
// on the server portion only (hostname + port) so this Map collects regions by server.
Map<HRegionLocation, MultiAction<Row>> actionsByServer =
new HashMap<HRegionLocation, MultiAction<Row>>();
List<Action<Row>> retainedActions = new ArrayList<Action<Row>>(rows.size()); do {
// Wait until there is at least one slot for a new task.
waitForMaximumCurrentTasks(maxTotalConcurrentTasks - 1); // Remember the previous decisions about regions or region servers we put in the
// final multi.
Map<String, Boolean> regionIncluded = new HashMap<String, Boolean>();
Map<ServerName, Boolean> serverIncluded = new HashMap<ServerName, Boolean>(); int posInList = -1;
Iterator<? extends Row> it = rows.iterator();
while (it.hasNext()) {
Row r = it.next();
HRegionLocation loc = findDestLocation(r, 1, posInList); if (loc != null && canTakeOperation(loc, regionIncluded, serverIncluded)) {
// loc is null if there is an error such as meta not available.
Action<Row> action = new Action<Row>(r, ++posInList);
retainedActions.add(action);
addAction(loc, action, actionsByServer);
it.remove();
}
} } while (retainedActions.isEmpty() && atLeastOne && !hasError()); HConnectionManager.ServerErrorTracker errorsByServer = createServerErrorTracker();
sendMultiAction(retainedActions, actionsByServer, 1, errorsByServer);
}

定位row找到loc(HRegionLocation)

HRegionLocation loc = findDestLocation(r, 1, posInList);

按region聚合action:addAction(loc, action, actionsByServer);

//Group the actions per region server
private void addAction(HRegionLocation loc, Action<Row> action, Map<HRegionLocation,
MultiAction<Row>> actionsByServer) {
final byte[] regionName = loc.getRegionInfo().getRegionName();
MultiAction<Row> multiAction = actionsByServer.get(loc);
if (multiAction == null) {
multiAction = new MultiAction<Row>();
actionsByServer.put(loc, multiAction);
} multiAction.add(regionName, action);
}

然后是sendMultiAction()

public void sendMultiAction(final List<Action<Row>> initialActions,
Map<HRegionLocation, MultiAction<Row>> actionsByServer,
final int numAttempt,
final HConnectionManager.ServerErrorTracker errorsByServer) {
// Send the queries and add them to the inProgress list
// This iteration is by server (the HRegionLocation comparator is by server portion only).
for (Map.Entry<HRegionLocation, MultiAction<Row>> e : actionsByServer.entrySet()) {
final HRegionLocation loc = e.getKey();
final MultiAction<Row> multiAction = e.getValue();
incTaskCounters(multiAction.getRegions(), loc.getServerName());
Runnable runnable = Trace.wrap("AsyncProcess.sendMultiAction", new Runnable() {
@Override
public void run() {
MultiResponse res;
try {
MultiServerCallable<Row> callable = createCallable(loc, multiAction);
try {
res = createCaller(callable).callWithoutRetries(callable);
} catch (IOException e) {
LOG.warn("Call to " + loc.getServerName() + " failed numAttempt=" + numAttempt +
", resubmitting all since not sure where we are at", e);
resubmitAll(initialActions, multiAction, loc, numAttempt + 1, e, errorsByServer);
return;
} receiveMultiAction(initialActions, multiAction, loc, res, numAttempt, errorsByServer);
} finally {
decTaskCounters(multiAction.getRegions(), loc.getServerName());
}
}
}); try {
this.pool.submit(runnable);
} catch (RejectedExecutionException ree) {
// This should never happen. But as the pool is provided by the end user, let's secure
// this a little.
decTaskCounters(multiAction.getRegions(), loc.getServerName());
LOG.warn("The task was rejected by the pool. This is unexpected." +
" Server is " + loc.getServerName(), ree);
// We're likely to fail again, but this will increment the attempt counter, so it will
// finish.
resubmitAll(initialActions, multiAction, loc, numAttempt + 1, ree, errorsByServer);
}
}
}

这里有几个都不是非常懂的样子

1。resubmitAll

2,receiveMultiAction

3,createCaller

直到后面在callable里的call方法里看到了responseProto = getStub().multi(controller, requestProto);这不是HRegionServer.multi()

先看callable吧,其它的慢慢再看。这个后面有 this.pool.submit(runnable)来提交运行的

callable创建

  protected MultiServerCallable<Row> createCallable(final HRegionLocation location,
final MultiAction<Row> multi) {
return new MultiServerCallable<Row>(hConnection, tableName, location, multi);
}

call()方法

  public MultiResponse call() throws IOException {
int countOfActions = this.multiAction.size();
if (countOfActions <= 0) throw new DoNotRetryIOException("No Actions");
MultiRequest.Builder multiRequestBuilder = MultiRequest.newBuilder();
List<CellScannable> cells = null;
// The multi object is a list of Actions by region. Iterate by region.
for (Map.Entry<byte[], List<Action<R>>> e: this.multiAction.actions.entrySet()) {
final byte [] regionName = e.getKey();
final List<Action<R>> actions = e.getValue();
RegionAction.Builder regionActionBuilder;
if (this.cellBlock) {
// Presize. Presume at least a KV per Action. There are likely more.
if (cells == null) cells = new ArrayList<CellScannable>(countOfActions);
// Send data in cellblocks. The call to buildNoDataMultiRequest will skip RowMutations.
// They have already been handled above. Guess at count of cells
regionActionBuilder = RequestConverter.buildNoDataRegionAction(regionName, actions, cells);
} else {
regionActionBuilder = RequestConverter.buildRegionAction(regionName, actions);
}
multiRequestBuilder.addRegionAction(regionActionBuilder.build());
}
// Controller optionally carries cell data over the proxy/service boundary and also
// optionally ferries cell response data back out again.
PayloadCarryingRpcController controller = new PayloadCarryingRpcController(cells);
controller.setPriority(getTableName());
ClientProtos.MultiResponse responseProto;
ClientProtos.MultiRequest requestProto = multiRequestBuilder.build();
try {
responseProto = getStub().multi(controller, requestProto);
} catch (ServiceException e) {
return createAllFailedResponse(requestProto, ProtobufUtil.getRemoteException(e));
}
return ResponseConverter.getResults(requestProto, responseProto, controller.cellScanner());
}

这里主要看HRegionServer.multi()

  public MultiResponse multi(final RpcController rpcc, final MultiRequest request)
throws ServiceException {
// rpc controller is how we bring in data via the back door; it is unprotobuf'ed data.
// It is also the conduit via which we pass back data.
PayloadCarryingRpcController controller = (PayloadCarryingRpcController)rpcc;
CellScanner cellScanner = controller != null? controller.cellScanner(): null;
if (controller != null) controller.setCellScanner(null);
List<CellScannable> cellsToReturn = null;
MultiResponse.Builder responseBuilder = MultiResponse.newBuilder(); for (RegionAction regionAction : request.getRegionActionList()) {
this.requestCount.add(regionAction.getActionCount());
RegionActionResult.Builder regionActionResultBuilder = RegionActionResult.newBuilder();
HRegion region;
try {
region = getRegion(regionAction.getRegion());
} catch (IOException e) {
regionActionResultBuilder.setException(ResponseConverter.buildException(e));
responseBuilder.addRegionActionResult(regionActionResultBuilder.build());
continue; // For this region it's a failure.
} if (regionAction.hasAtomic() && regionAction.getAtomic()) {
// How does this call happen? It may need some work to play well w/ the surroundings.
// Need to return an item per Action along w/ Action index. TODO.
try {
mutateRows(region, regionAction.getActionList(), cellScanner);
} catch (IOException e) {
// As it's atomic, we may expect it's a global failure.
regionActionResultBuilder.setException(ResponseConverter.buildException(e));
}
} else {
// doNonAtomicRegionMutation manages the exception internally
cellsToReturn = doNonAtomicRegionMutation(region, regionAction, cellScanner,
regionActionResultBuilder, cellsToReturn);
}
responseBuilder.addRegionActionResult(regionActionResultBuilder.build());
}
// Load the controller with the Cells to return.
if (cellsToReturn != null && !cellsToReturn.isEmpty() && controller != null) {
controller.setCellScanner(CellUtil.createCellScanner(cellsToReturn));
}
return responseBuilder.build();
}

主要代码。其它都是build PB

 region = getRegion(regionAction.getRegion());
mutateRows(region, regionAction.getActionList(), cellScanner);
cellsToReturn = doNonAtomicRegionMutation(region, regionAction, cellScanner,
regionActionResultBuilder, cellsToReturn);

当中mutateRows()是PUT和DELETE相关的

  protected void mutateRows(final HRegion region, final List<ClientProtos.Action> actions,
final CellScanner cellScanner)
throws IOException {
if (!region.getRegionInfo().isMetaTable()) {
cacheFlusher.reclaimMemStoreMemory();
}
RowMutations rm = null;
for (ClientProtos.Action action: actions) {
if (action.hasGet()) {
throw new DoNotRetryIOException("Atomic put and/or delete only, not a Get=" +
action.getGet());
}
MutationType type = action.getMutation().getMutateType();
if (rm == null) {
rm = new RowMutations(action.getMutation().getRow().toByteArray());
}
switch (type) {
case PUT:
rm.add(ProtobufUtil.toPut(action.getMutation(), cellScanner));
break;
case DELETE:
rm.add(ProtobufUtil.toDelete(action.getMutation(), cellScanner));
break;
default:
throw new DoNotRetryIOException("Atomic put and/or delete only, not " + type.name());
}
}
region.mutateRow(rm);
}

hbase0.96 put流程 源码分析的更多相关文章

  1. Django rest framework 的认证流程(源码分析)

    一.基本流程举例: urlpatterns = [ url(r'^admin/', admin.site.urls), url(r'^users/', views.HostView.as_view() ...

  2. springMvc的执行流程(源码分析)

    1.在springMvc中负责处理请求的类为DispatcherServlet,这个类与我们传统的Servlet是一样的.我们来看看它的继承图 2. 我们发现DispatcherServlet也继承了 ...

  3. Spring Securtiy 认证流程(源码分析)

    当用 Spring Security 框架进行认证时,你可能会遇到这样的问题: 你输入的用户名或密码不管是空还是错误,它的错误信息都是 Bad credentials. 那么如果你想根据不同的情况给出 ...

  4. springmvc执行流程 源码分析

    进入DispatcherServlet 执行onRefresh,然后执行初始化方法initStrategies.然后调用doService——>doDispatch. 根据继承关系执行Servl ...

  5. drf的基本使用、APIView源码分析和CBV源码拓展

    cbv源码拓展 扩展,如果我在Book视图类中重写dispatch方法 -可以实现,在get,post方法执行之前或者之后执行代码,完成类似装饰器的效果 def dispatch(self, requ ...

  6. ThreadPoolExecutor源码分析-面试问烂了的Java线程池执行流程,如果要问你具体的执行细节,你还会吗?

    Java版本:8u261. 对于Java中的线程池,面试问的最多的就是线程池中各个参数的含义,又或者是线程池执行的流程,彷佛这已成为了固定的模式与套路.但是假如我是面试官,现在我想问一些更细致的问题, ...

  7. zookeeper源码分析之五服务端(集群leader)处理请求流程

    leader的实现类为LeaderZooKeeperServer,它间接继承自标准ZookeeperServer.它规定了请求到达leader时需要经历的路径: PrepRequestProcesso ...

  8. zookeeper源码分析之四服务端(单机)处理请求流程

    上文: zookeeper源码分析之一服务端启动过程 中,我们介绍了zookeeper服务器的启动过程,其中单机是ZookeeperServer启动,集群使用QuorumPeer启动,那么这次我们分析 ...

  9. zookeeper源码分析之三客户端发送请求流程

    znode 可以被监控,包括这个目录节点中存储的数据的修改,子节点目录的变化等,一旦变化可以通知设置监控的客户端,这个功能是zookeeper对于应用最重要的特性,通过这个特性可以实现的功能包括配置的 ...

随机推荐

  1. 分享一个自用的 Inno Setup 软件打包脚本

    此脚本支持打包mysql.安装mysql服务.安装windows服务.操作ini文件.操作注册表.高效压缩文件等功能,基本能满足常用的软件打包需求. ;定义各种常量 #define MyAppName ...

  2. qutIm编译

    官网:http://www.qutim.org/ 原文地址:http://wiki.qutim.org/en/building_from_git 依赖: Qt4-dev 4.7:http://qt-p ...

  3. XDocument读取xml的所有元素以及XPath语法

    原文  http://www.cnblogs.com/xxyishutong/p/3326375.html <?xml version="1.0" encoding=&quo ...

  4. JAVA代码静态检测之PMD

    今天再次想启动Java代码静态检测工具的利用问题,主要再次尝试用了PMD,发现不少代码编码规范问题和好的代码建议,并学到不少自己之前没有注意到的Java方便的基础知识,感觉很不错,把相关明白的好的规则 ...

  5. docker基于 aufs 文件系统

    docker的核心功能就是容器版本管理,在容器层实现了CVS版本管理,比如git那种可以commit/roll back 而AUFS就是其实现基础 AUFS可以实现文件系统的snapshot,这样对f ...

  6. AS3游戏中可视对象上限及位图相关的内存消耗实测

    前些天连续做了一些测试,以加深对AS3的掌握和在项目中对 游戏 性能.效率优化方面的一些处理,有很多测试实际意义不大,都不过是证明一些猜想是正确的,除此没有什么. 但前天进行的一系列测试中,有一些对游 ...

  7. ListView分割线

    在开发中遇到需要ListView 中每一个条目后面都有个分隔线,但是总是发现最后一个条目后面没有分隔符,后来查到原因,是因为ListView的layout_height=“wrap_content” ...

  8. k8s 集群基本概念

    一.概述: kubernetes是google开源的容器集群管理系统,提供应用部署.维护.扩展机制等功能,利用kubernetes能方便管理跨集群运行容器化的应用,简称:k8s(k与s之间有8个字母) ...

  9. EasyUI中datagrid实现显示、增加、 删除、 修改、 查询操作(后台代码C#)

    菜鸟进入,高手请绕道! +++++++++++++++++++++++++++++++++++++++ 一.数据的显示 1新建HtmlPage2.html页面,引入相关文件.如下所示 <scri ...

  10. HTML系列(四):编辑图像

    一.图像的基本概念 1.矢量图:文件占用空间小,放大后图像不会失真,和分辨率无关.适用于图形设计.文字设计.标志设计.版式设计等. 2.位图:由像素点组成,文件较大,放大和缩小图像会失真. 3.有损压 ...