HBase Compact

Region Compact请求是在Region MemStore Flush之后被触发的：

boolean shouldCompact = region.flushcache();

// We just want to check the size
boolean shouldSplit = region.checkSplit() != null;

if (shouldSplit) {
    this.server.compactSplitThread.requestSplit(region);
} else if (shouldCompact) {
    server.compactSplitThread.requestCompaction(region, getName());
}

server.getMetrics().addFlush(region.getRecentFlushInfo());

region.flushcache方法负责该Region实例的Flush操作，该操作的返回值shouldCompact如果为true，则表示该Region达到了Region Compact的要求，但此时并不一定会触发Compact请求，因为shouldCompact之后会有shouldSplit的判断，只有当shouldSplit为false且shouldCompact为true时才会触发Compact请求。

注：某一Region的Flush、Split、Compact是分别在不同的线程中进行的，三者之间的协调会在后续详细介绍，在此仅仅关注Compact的触发条件，以及Compact进行时如何选取StoreFile进行合并即可。

Region Compact触发（即shouldCompact为true）的判断条件是该Region中各个Store中的StoreFile数目：

/**
 * See if there's too much store files in this store
 *
 * @return true if number of store files is greater than the number defined
 *         in minFilesToCompact
 */
public boolean needsCompaction() {
    return (storefiles.size() - filesCompacting.size()) > minFilesToCompact;
}

在某一Region做Flush操作时，内部是逐个对该Region的多个Store依次进行Flush的，每一个Store Flush完成之后，就会调用上述代码进行判断：

storefiles：表示Store Flush之后，StoreFile的总数目；

filesCompacting：表示Store中正处于Compact状态的StoreFile的数目，即表示该Store中的某些StoreFile已被选择进行Compact，这次判断需要忽略这些StoreFile；

minFilesToCompact：表示进行Store Compact时Store中的StoreFile数目（不包含已被选择进行Compact的StoreFile）所需要达到的下限值，它的值是由以下代码计算而来的：

// By default, compact if storefile.count >= minFilesToCompact
this.minFilesToCompact = Math.max(2, conf.getInt(
        "hbase.hstore.compaction.min",
        /* old name */conf
            .getInt("hbase.hstore.compactionThreshold", 3)));

注意：hbase.hstore.compactionThreshold是一个旧的配置项，当配置项中出现hbase.hstore.compaction.min时，它将失效。

由上所示可知，shouldCompact为true的前提条件是当前Region中的某个Store中的StoreFile数目需要达到下限要求（需要去除那些已处于合并状态的StoreFile），如果满足条件，则通过requestCompaction发出具体的Compact请求，requestCompaction有多个重载方法，最终执行流程会被导向下述这些重载方法：

@Override
public synchronized List<CompactionRequest> requestCompaction(
        final HRegion r, final String why, int pri,
        final List<CompactionRequest> requests) throws IOException {
    List<CompactionRequest> ret;

    // not a special compaction request, so make out own list
    if (requests == null) {
        ret = new ArrayList<CompactionRequest>(r.getStores().size());

        for (Store s : r.getStores().values()) {
            ret.add(requestCompaction(r, s, why, pri, null));
        }
    } else {
        ret = new ArrayList<CompactionRequest>(requests.size());

        for (CompactionRequest request : requests) {
            ret.add(requestCompaction(r, request.getStore(), why, pri,
                    request));
        }
    }

    return ret;
}

可以看出，实际的Compact请求也是以Store为单位发出的，如下代码所示：

@Override
public synchronized CompactionRequest requestCompaction(final HRegion r,
        final Store s, final String why, int priority,
        CompactionRequest request) throws IOException {
    if (this.server.isStopped()) {
        return null;
    }

    CompactionRequest cr = s.requestCompaction(priority, request);

    if (cr != null) {
        cr.setServer(server);

        if (priority != Store.NO_PRIORITY) {
            cr.setPriority(priority);
        }

        ThreadPoolExecutor pool = s.throttleCompaction(cr.getSize()) ? largeCompactions
                : smallCompactions;

        pool.execute(cr);

        if (LOG.isDebugEnabled()) {
            String type = (pool == smallCompactions) ? "Small " : "Large ";

            LOG.debug(type
                    + "Compaction requested: "
                    + cr
                    + (why != null && !why.isEmpty() ? "; Because: " + why
                            : "") + "; " + this);
        }
    } else {
        if (LOG.isDebugEnabled()) {
            LOG.debug("Not compacting " + r.getRegionNameAsString()
                    + " because compaction request was cancelled");
        }
    }

    return cr;
}

首先会根据Store的状态形成一个CompactionRequest对象（实现了Runnable接口，所谓的请求实际上就是一个Runnable对象，以线程的形式执行），然后根据当前Store需要进行Compact操作的StoreFile的总大小（不是文件数目）判断是何种类型的Compact（LargeCompaction或者SmallCompaction），从而将CompactionRequest提交至不同的线程池中执行。

LargeCompaction与SmallCompaction的选择过程如下：

boolean throttleCompaction(long compactionSize) {
    long throttlePoint = conf.getLong(
            "hbase.regionserver.thread.compaction.throttle", 2
                    * this.minFilesToCompact
                    * this.region.memstoreFlushSize);

    return compactionSize > throttlePoint;
}

至此，CompactRequest请求已被提交至相应的线程池中，根据线程池的执行策略，在适当的时机被执行。

CompactionRequest

如何根据Store的状态生成相应的CompactionRequest对象，即选择Store中的哪些StoreFile进行Compact，该过程是由Store requestCompaction方法完成的，下面对该方法的核心代码进行讲述。

代码主要是在一个同步代码块中完成的，

synchronized (filesCompacting) {
    ......
}

随着MemStore的不断Flush操作，可能会造成同一个Region的多次Compact Request，为了避免同一StoreFile出现在多个Compact中，所以需要对filesCompacting（表示已被选取进行Compact的StoreFile）进行互斥访问。

接下来的所有代码都是出现在上述同步代码块中的。

// candidates = all storefiles not already in compaction queue
List<StoreFile> candidates = Lists.newArrayList(storefiles);

选取Store中的所有StoreFile作为备选对象，storefiles是依据StoreFile.Comparators.FLUSH_TIME进行排序的，此时candidates以及后续的filesCompacting也是有序的（older –> newer）。

if (!filesCompacting.isEmpty()) {
    // exclude all files older than the newest file we're
    // currently
    // compacting. this allows us to preserve contiguity
    // (HBASE-2856)
    StoreFile last = filesCompacting
            .get(filesCompacting.size() - 1);

    int idx = candidates.indexOf(last);

    Preconditions.checkArgument(idx != -1);

    candidates.subList(0, idx + 1).clear();
}

如果filesCompacting不为空，则需要从candidates中移除一些StoreFile：

（1）选中filesCompacting中的最后一个StoreFile保存至变量last中，即filesCompacting中FlushTime最新的那个StoreFile；

（2）移除candidates中所有FlushTime比last旧的StoreFile。

这样做的目的主要是为了Compact操作的持续性，即每次尽可以多的合并新产生的StoreFile，不要重复去Compact那些已经被合并过或正在进行Compact的StoreFile。

boolean override = false;

if (region.getCoprocessorHost() != null) {
    override = region.getCoprocessorHost().preCompactSelection(
            this, candidates, request);
}

CompactSelection filesToCompact;

if (override) {
    // coprocessor is overriding normal file selection
    filesToCompact = new CompactSelection(conf, candidates);
} else {
    filesToCompact = compactSelection(candidates, priority);
}

if (region.getCoprocessorHost() != null) {
    region.getCoprocessorHost().postCompactSelection(
            this,
            ImmutableList.copyOf(filesToCompact
                .getFilesToCompact()), request);
}

这里不讨论Coprocessor对Compact的影响，所以override为false，执行compactSelection算法（详细算法描述见后）进行选取，选取结果会被保存至filesToCompact中。

// no files to compact
if (filesToCompact.getFilesToCompact().isEmpty()) {
    return null;
}

经过相应算法选取之后，如果选取结果为空，则表示不需要进行Compact，返回null即可。

// basic sanity check: do not try to compact the same StoreFile
// twice.
if (!Collections.disjoint(filesCompacting,
        filesToCompact.getFilesToCompact())) {
    // TODO: change this from an IAE to LOG.error after
    // sufficient testing
    Preconditions.checkArgument(false, "%s overlaps with %s",
            filesToCompact, filesCompacting);
}

这是一个对选取结果进行核验的工作，主要是为了保证选取结果不会出现在filesCompacting中，即不会对同一个StoreFile进行两次Compact操作。

filesCompacting.addAll(filesToCompact.getFilesToCompact());

Collections.sort(filesCompacting,
        StoreFile.Comparators.FLUSH_TIME);

将选取结果添加至filesCompacting中，并对更新后的filesCompacting依据FlushTime重新排序。

// major compaction iff all StoreFiles are included
boolean isMajor = (filesToCompact.getFilesToCompact().size() == this.storefiles
        .size());

if (isMajor) {
    // since we're enqueuing a major, update the compaction wait
    // interval
    this.forceMajor = false;
}

如果选取结果包含该Store中所有的StoreFile，则表示我们应该进行一个Major Compaction，并取消Force Major Compaction。

// everything went better than expected. create a compaction
// request
int pri = getCompactPriority(priority);

计算本次Compact的优先级：

/**
* @return The priority that this store should have in the compaction queue
 * @param priority
 */
public int getCompactPriority(int priority) {
    // If this is a user-requested compaction, leave this at the highest
    // priority
    if (priority == PRIORITY_USER) {
        return PRIORITY_USER;
    } else {
        return this.blockingStoreFileCount - this.storefiles.size();
    }
}

如果Compact Request是由用户发起的，则需要给该Compaction赋于最高的优先级PRIORITY_USER（1），否则需要进行差值计算得出，其中blockingStoreFileCount来源于如下代码：

this.blockingStoreFileCount = conf.getInt(
    "hbase.hstore.blockingStoreFiles", 7);

以下代码开始生成具体的CompactionRequest的对象，在我们讲述的流程中request为null，所以通过相应的构造函数创建对象即可。

// not a special compaction request, so we need to make one
if (request == null) {
    request = new CompactionRequest(region, this,
            filesToCompact, isMajor, pri);
} else {
    // update the request with what the system thinks the
    // request should be
    // its up to the request if it wants to listen
    request.setSelection(filesToCompact);

    request.setIsMajor(isMajor);

    request.setPriority(pri);
}

代码至此完成了一个CompactRequest的形成。

 

CompactSelection

该算法由方法compactSelection实现：

/**
 * Algorithm to choose which files to compact
 *
 * Configuration knobs:
 *
 * "hbase.hstore.compaction.ratio" normal case: minor compact when file <=
 * sum(smaller_files) * ratio
 *
 * "hbase.hstore.compaction.min.size" unconditionally compact individual
 * files below this size
 *
 * "hbase.hstore.compaction.max.size" never compact individual files above
 * this size (unless splitting)
 *
 * "hbase.hstore.compaction.min" min files needed to minor compact
 *
 * "hbase.hstore.compaction.max" max files to compact at once (avoids OOM)
 *
 * @param candidates
 *            candidate files, ordered from oldest to newest
 * @return subset copy of candidate list that meets compaction criteria
 * @throws IOException
 */
CompactSelection compactSelection(List<StoreFile> candidates, int priority)
    throws IOException {
    ......
}

方法开始处有如下注释:

// ASSUMPTION!!! filesCompacting is locked when calling this function

/*
 * normal skew:
 *
 * older ----> newer _ | | _ | | | | _ --|-|- |-|-
 * |-|---_-------_------- minCompactSize | | | | | | | | _ | | | | | | |
 * | | | | | | | | | | | | | | | | | | |
 */

该算法的选取主要从旧到新依次进行，minCompactSize 的应用见后面代码所示。

CompactSelection compactSelection = new CompactSelection(conf,
    candidates);

创建一个CompactSelection对象，在构造方法中主要是初始化了一些参数。

boolean forcemajor = this.forceMajor && filesCompacting.isEmpty();

根据forceMajor（在compactSelection方法调用之前已经计算出该值）与filesCompacting的状态计算forcemajor的值。如果forecemajor为false则执行下述代码：

if (!forcemajor) {
    // Delete the expired store files before the compaction selection.
    if (conf.getBoolean("hbase.store.delete.expired.storefile", true)
            && (ttl != Long.MAX_VALUE)
            && (this.scanInfo.minVersions == 0)) {
        CompactSelection expiredSelection = compactSelection
                .selectExpiredStoreFilesToCompact(EnvironmentEdgeManager
                    .currentTimeMillis() - this.ttl);

        // If there is any expired store files, delete them by
        // compaction.
        if (expiredSelection != null) {
            return expiredSelection;
        }
    }

    // do not compact old files above a configurable threshold
    // save all references. we MUST compact them
    int pos = 0;

    while (pos < compactSelection.getFilesToCompact().size()
        && compactSelection.getFilesToCompact().get(pos)
            .getReader().length() > maxCompactSize
        && !compactSelection.getFilesToCompact().get(pos)
            .isReference()) {
        ++pos;
    }

    if (pos != 0) {
        compactSelection.clearSubList(0, pos);
    }
}

其中

// Delete the expired store files before the compaction selection.
if (conf.getBoolean("hbase.store.delete.expired.storefile", true)
        && (ttl != Long.MAX_VALUE)
        && (this.scanInfo.minVersions == 0)) {
    CompactSelection expiredSelection = compactSelection
            .selectExpiredStoreFilesToCompact(EnvironmentEdgeManager
                    .currentTimeMillis() - this.ttl);

    // If there is any expired store files, delete them by
    // compaction.
    if (expiredSelection != null) {
        return expiredSelection;
    }
}

这部分代码主要用于删除那些过期的StoreFile，如果存在有过期的StoreFile，则算法的选取结果即为这些过期的StoreFile，不再需要进行进一步的选取，在此先忽略这一步。

// do not compact old files above a configurable threshold
// save all references. we MUST compact them
int pos = 0;

while (pos < compactSelection.getFilesToCompact().size()
        && compactSelection.getFilesToCompact().get(pos)
                .getReader().length() > maxCompactSize
        && !compactSelection.getFilesToCompact().get(pos)
                .isReference()) {
    ++pos;
}

if (pos != 0) {
    compactSelection.clearSubList(0, pos);
}

compactSelection.getFilesToCompact()的返回值即为candidates，该代码的作用是从前到后淘汰一些文件大小超过配置大小的StoreFile，直至在此过程中遇到下述两种情况：

（1）某StoreFile的文件大小小于或等于maxCompactSize（hbase.hstore.compaction.max.size）；

（2）某StoreFile的文件类型为Reference（该文件类型在Split的过程中产生）。

if (compactSelection.getFilesToCompact().isEmpty()) {
    LOG.debug(this.getHRegionInfo().getEncodedName() + " - " + this
            + ": no store files to compact");

    compactSelection.emptyFileList();

    return compactSelection;
}

如果经过前一步的淘汰过程之后，compactSelection.getFilesToCompact()的返回结果为空，则表示没有相应的StoreFile可以进行Compact，选取结束，直接返回即可。

// Force a major compaction if this is a user-requested major
// compaction,
// or if we do not have too many files to compact and this was requested
// as a major compaction
boolean majorcompaction = (forcemajor && priority == PRIORITY_USER)
    || (forcemajor || isMajorCompaction(compactSelection
        .getFilesToCompact()))
    && (compactSelection.getFilesToCompact().size() < this.maxFilesToCompact);

majorcompaction为true，需要达到下述两个条件之一：

（1）用户主动请求进行Major Compaction；

（2）forcemajor为true或者满足系统主动进行Major Compaction的条件（主要根据StoreFile时间戳进行判断），而且此时待选取的StoreFile数目没有超过临界值（maxFilesToCompact：hbase.hstore.compaction.max）。

关于用户主动请求或系统主动进行Major Compaction的情况，后续再进行讨论。

接下来的代码会根据条件的不同出现两个大的代码块，分别进行讲述。

if (!majorcompaction
    && !hasReferences(compactSelection.getFilesToCompact())) {
    ......
} else {
    ......
}

（1）如果majorcompaction为false且剩余的待选取StoreFile中不包含Reference类型的文件，则执行如下代码：

// we're doing a minor compaction, let's see what files are
// applicable
int start = 0;

double r = compactSelection.getCompactSelectionRatio();

r的值即为hbase.hstore.compaction.ratio，不考虑配置hbase.offpeak.start.hour、hbase.offpeak.end.hour情况。

// remove bulk import files that request to be excluded from minors
compactSelection.getFilesToCompact().removeAll(
        Collections2.filter(compactSelection.getFilesToCompact(),
                new Predicate<StoreFile>() {
                    public boolean apply(StoreFile input) {
                        return input.excludeFromMinorCompaction();
                    }
                }));

在待选取的StoreFile中移除那些已经被设置从Minor Compaction中移除的StoreFile。

// skip selection algorithm if we don't have enough files
if (compactSelection.getFilesToCompact().size() < this.minFilesToCompact) {
    if (LOG.isDebugEnabled()) {
        LOG.debug("Not compacting files because we only have "
                + compactSelection.getFilesToCompact().size()
                + " files ready for compaction.  Need "
                + this.minFilesToCompact + " to initiate.");
    }

    compactSelection.emptyFileList();

    return compactSelection;
}

如果此时我们选取的StoreFile数目小于minFilesToCompact（配置时注意该值不包含等于的情况），则本次Compact的操作取消，直接返回即可，其中minFilesToCompact的计算代码如下：

// By default, compact if storefile.count >= minFilesToCompact
this.minFilesToCompact = Math.max(2, conf.getInt(
    "hbase.hstore.compaction.min",
    /* old name */conf
        .getInt("hbase.hstore.compactionThreshold", 3)));

继续进行选取算法。

// get store file sizes for incremental compacting selection.
int countOfFiles = compactSelection.getFilesToCompact().size();

long[] fileSizes = new long[countOfFiles];

long[] sumSize = new long[countOfFiles];

for (int i = countOfFiles - 1; i >= 0; --i) {
    StoreFile file = compactSelection.getFilesToCompact().get(i);

    fileSizes[i] = file.getReader().length();

    // calculate the sum of fileSizes[i,i+maxFilesToCompact-1) for
    // algo
    int tooFar = i + this.maxFilesToCompact - 1;

    sumSize[i] = fileSizes[i]
            + ((i + 1 < countOfFiles) ? sumSize[i + 1] : 0)
            - ((tooFar < countOfFiles) ? fileSizes[tooFar] : 0);
}

上述代码实际是做一些准备工作，主要生成了两个重要的数组：fileSizes、sumSize。

fileSizes：依次保存着各个StoreFile的文件大小；

sumSize：对应fileSizes，依次保存着fileSizes[i,i+maxFilesToCompact-1)之和，即sumSize[i]中保存着compactSelection.getFilesToCompact()中第i个StoreFile（包含）之后最多maxFilesToCompact个StoreFile的总大小。

根据这两个数组，继续选举算法：

/*
 * Start at the oldest file and stop when you find the first file
 * that meets compaction criteria:
 *
 * (1) a recently-flushed, small file (i.e. <= minCompactSize) OR
 * (2) within the compactRatio of sum(newer_files)
 *
 * Given normal skew, any newer files will also meet this criteria
 *
 * Additional Note: If fileSizes.size() >> maxFilesToCompact, we
* will recurse on compact(). Consider the oldest files first to
 * avoid a situation where we always compact [end-threshold,end).
 * Then, the last file becomes an aggregate of the previous
 * compactions.
 */
while (countOfFiles - start >= this.minFilesToCompact
        && fileSizes[start] > Math.max(minCompactSize,
                (long) (sumSize[start + 1] * r))) {
    ++start;
}

int end = Math.min(countOfFiles, start + this.maxFilesToCompact);

long totalSize = fileSizes[start]
        + ((start + 1 < countOfFiles) ? sumSize[start + 1] : 0);

compactSelection = compactSelection.getSubList(start, end);

从前到后（根据FlushTime从旧到新）依次淘汰相应的StoreFile，直至遇到某一StoreFile的大小满足条件：

fileSizes[start] <= Math.max(minCompactSize, (long) (sumSize[start + 1] * r))

minCompactSize：hbase.hstore.compaction.min.size

理解其中的一种情况：fileSizes[start] > (long) (sumSize[start + 1] * r) > minCompactSize，它表示当前StoreFile的大小比它之后maxFilesToCompact范围内的所有StoreFile大小之和还要大，我们认为当前StoreFile太大，应淘汰掉，优先合并它之后那些较小的StoreFile。

根据注释可知，HBase希望最近Flush、Small的StoreFile文件大小满足下述两个条件之一：

（1）小于等于minCompactSize；

（2）小于等于the compactRatio of sum(newer_files)。

上述代码最后会更新compactSelection对象的状态。

// if we don't have enough files to compact, just wait
if (compactSelection.getFilesToCompact().size() < this.minFilesToCompact) {
    if (LOG.isDebugEnabled()) {
        LOG.debug("Skipped compaction of " + this + ".  Only "
                + (end - start) + " file(s) of size "
                + StringUtils.humanReadableInt(totalSize)
                + " have met compaction criteria.");
    }

    compactSelection.emptyFileList();

    return compactSelection;
}

如果此时没有足够的StoreFile进行Compact（StoreFile数目小于minFilesToCompact），则跳过这次Compact，直接返回。

（2）如果majorcompaction为true或者待选取的StoreFile中包含Reference文件，则执行如下代码：

if (majorcompaction) {
    if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
        LOG.debug("Warning, compacting more than "
            + this.maxFilesToCompact
            + " files, probably because of a user-requested major compaction");

        if (priority != PRIORITY_USER) {
            LOG.error("Compacting more than max files on a non user-requested compaction");
        }
    }
} else if (compactSelection.getFilesToCompact().size() > this.maxFilesToCompact) {
    // all files included in this compaction, up to max
    int pastMax = compactSelection.getFilesToCompact().size()
            - this.maxFilesToCompact;

    compactSelection.getFilesToCompact().subList(0, pastMax)
            .clear();
}

如果majorcompaction为true，仅需要在选取进行Compact的StoreFile数目大于maxFilesToCompact（hbase.hstore.compaction.max）时，打印一些信息即可；

如果majorcompaction为false，且在选取进行Compact的StoreFile数目大于maxFilesToCompact（hbase.hstore.compaction.max）时，移除掉后面多余的StoreFile。

return compactSelection;

至此，选举算法结束。