企业搜索引擎开发之连接器connector(二十九)
在哪里调用监控器管理对象snapshotRepositoryMonitorManager的start方法及stop方法,然后又在哪里调用CheckpointAndChangeQueue对象的resume方法获取List<CheckpointAndChange> guaranteedChanges集合
下面跟踪到DiffingConnectorTraversalManager类的相关方法,在该类实现的方法中,调用了监控器管理对象snapshotRepositoryMonitorManager的相关方法实现对其操作
private final DocumentSnapshotRepositoryMonitorManager
snapshotRepositoryMonitorManager;
private final TraversalContextManager traversalContextManager;
/**
* Boolean to mark TraversalManager as invalid.
* It's possible for Connector Manager to keep a reference to
* an outdated TraversalManager (after a new one has been given
* previous TraversalManagers are invalid to use).
*/
private boolean isActive = true; /**
* Creates a {@link DiffingConnectorTraversalManager}.
*
* @param snapshotRepositoryMonitorManager the
* {@link DocumentSnapshotRepositoryMonitorManager}
* for use accessing a {@link ChangeSource}
* @param traversalContextManager {@link TraversalContextManager}
* that holds the current {@link TraversalContext}
*/
public DiffingConnectorTraversalManager(
DocumentSnapshotRepositoryMonitorManager snapshotRepositoryMonitorManager,
TraversalContextManager traversalContextManager) {
this.snapshotRepositoryMonitorManager = snapshotRepositoryMonitorManager;
this.traversalContextManager = traversalContextManager;
}
resumeTraversal方法启动监视器管理对象snapshotRepositoryMonitorManager,并返回DocumentList集合
/* @Override */
public synchronized DocumentList resumeTraversal(String checkpoint)
throws RepositoryException {
/* Exhaustive list of method's use:
resumeTraversal(null) from startTraversal:
monitors get started from null
resumeTraversal(null) from Connector Manager sometime after startTraversal:
monitors already started from previous resumeTraversal call
resumeTraversal(cp) from Connector Manager without a startTraversal:
means there was a shutdown or turn off
monitors get started from cp; should use state
resumeTraversal(cp) from Connector Manager sometime after some uses:
is most common case; roll
*/
if (isActive()) {
//启动snapshotRepositoryMonitorManager
if (!snapshotRepositoryMonitorManager.isRunning()) {
snapshotRepositoryMonitorManager.start(checkpoint);
}
return newDocumentList(checkpoint);
} else {
throw new RepositoryException(
"Inactive FileTraversalManager referanced.");
}
}
进一步调用newDocumentList方法返回DocumentList集合
private DocumentList newDocumentList(String checkpoint)
throws RepositoryException {
//获取队列 CheckpointAndChangeQueue(队列 CheckpointAndChangeQueue只由snapshotRepositoryMonitorManager引用)
CheckpointAndChangeQueue checkpointAndChangeQueue =
snapshotRepositoryMonitorManager.getCheckpointAndChangeQueue(); try {
DiffingConnectorDocumentList documentList = new DiffingConnectorDocumentList(
checkpointAndChangeQueue,
CheckpointAndChangeQueue.initializeCheckpointStringIfNull(
checkpoint));
//Map<String, MonitorCheckpoint>
Map<String, MonitorCheckpoint> guaranteesMade =
checkpointAndChangeQueue.getMonitorRestartPoints(); snapshotRepositoryMonitorManager.acceptGuarantees(guaranteesMade); return new ConfirmActiveDocumentList(documentList);
} catch (IOException e) {
throw new RepositoryException("Failure when making DocumentList.", e);
}
}
DiffingConnectorDocumentList documentList对象的构造函数里面封装了CheckpointAndChangeQueue checkpointAndChangeQueue队列集合
DiffingConnectorDocumentList 类完整实现如下:
/**
* An implementation of {@link DocumentList} for the {@link DiffingConnector}.
*
* @since 2.8
*/
public class DiffingConnectorDocumentList implements DocumentList {
private final Iterator<CheckpointAndChange> checkpointAndChangeIterator;
private String checkpoint; /**
* Creates a document list that returns a batch of documents from the provided
* {@link CheckpointAndChangeQueue}.
*
* @param queue a CheckpointAndChangeQueue containing document changes
* @param checkpoint point into the change queue after which to start
* returning documents
* @throws IOException if persisting fails
*/
public DiffingConnectorDocumentList(CheckpointAndChangeQueue queue,
String checkpoint) throws IOException {
//CheckpointAndChangeQueue queued的resume方法获取List<CheckpointAndChange>
//本DocumentList批次数据已经加载于内存
List<CheckpointAndChange> guaranteedChanges = queue.resume(checkpoint);
checkpointAndChangeIterator = guaranteedChanges.iterator();
this.checkpoint = checkpoint;
} /**
* 调用方获取该状态并持久化,迭代完毕即为最后的checkpoint
*/
/* @Override */
public String checkpoint() {
return checkpoint;
} /* @Override */
public Document nextDocument() throws RepositoryException {
if (checkpointAndChangeIterator.hasNext()) {
CheckpointAndChange checkpointAndChange =
checkpointAndChangeIterator.next();
//更新checkpoint
checkpoint = checkpointAndChange.getCheckpoint().toString();
return checkpointAndChange.getChange().getDocumentHandle().getDocument();
} else {
return null;
}
}
}
在其构造方法中调用参数CheckpointAndChangeQueue queue的resume方法获取List<CheckpointAndChange> guaranteedChanges,在其nextDocument()方法中通过迭代获取CheckpointAndChange checkpointAndChange对象,同时更新checkpoint状态标识
最后获取与监视器关联的MonitorCheckpoint对象映射
//Map<String, MonitorCheckpoint>
Map<String, MonitorCheckpoint> guaranteesMade =
checkpointAndChangeQueue.getMonitorRestartPoints();
然后调用监控器管理对象snapshotRepositoryMonitorManager的acceptGuarantees方法,相应的监视器对象接收并确认MonitorCheckpoint对象
/**
* 监视器管理对象收到CheckpointAndChangeQueue对象反馈,分发给对应的监视器处理MonitorCheckpoint
*/
/* @Override */
public void acceptGuarantees(Map<String, MonitorCheckpoint> guarantees) {
for (Map.Entry<String, MonitorCheckpoint> entry : guarantees.entrySet()) {
String monitorName = entry.getKey();
MonitorCheckpoint checkpoint = entry.getValue();
DocumentSnapshotRepositoryMonitor monitor = fileSystemMonitorsByName.get(monitorName);
if (monitor != null) {
// Signal is asynch. Let monitor figure out how to use.
//回调
monitor.acceptGuarantee(checkpoint);
}
}
}
与仓库对象相对应的具体监视器接收确认
/**
* 监视器收到反馈 [MonitorCheckpoint接收确认]
* @param cp
*/
// Public for DocumentSnapshotRepositoryMonitorTest
@VisibleForTesting
public void acceptGuarantee(MonitorCheckpoint cp) {
snapshotStore.acceptGuarantee(cp);
guaranteeCheckpoint = cp;
}
仓库对应的存储对象处于处理链的末端
/**
* 反馈MonitorCheckpoint处理
* @param cp
*/
void acceptGuarantee(MonitorCheckpoint cp) {
long readSnapshotNumber = cp.getSnapshotNumber();
if (readSnapshotNumber < 0) {
throw new IllegalArgumentException("Received invalid snapshot in: " + cp);
}
if (oldestSnapshotToKeep > readSnapshotNumber) {
LOG.warning("Received an older snapshot than " + oldestSnapshotToKeep + ": " + cp);
} else {
oldestSnapshotToKeep = readSnapshotNumber;
}
}
---------------------------------------------------------------------------
本系列企业搜索引擎开发之连接器connector系本人原创
转载请注明出处 博客园 刺猬的温驯
本人邮箱: chenying998179@163#com (#改为.)
本文链接 http://www.cnblogs.com/chenying99/p/3789650.html
企业搜索引擎开发之连接器connector(二十九)的更多相关文章
- 企业搜索引擎开发之连接器connector(十九)
连接器是基于http协议通过推模式(push)向数据接收服务端推送数据,即xmlfeed格式数据(xml格式),其发送数据接口命名为Pusher Pusher接口定义了与发送数据相关的方法 publi ...
- 企业搜索引擎开发之连接器connector(十八)
创建并启动连接器实例之后,连接器就会基于Http协议向指定的数据接收服务器发送xmlfeed格式数据,我们可以通过配置http代理服务器抓取当前基于http协议格式的数据(或者也可以通过其他网络抓包工 ...
- 企业搜索引擎开发之连接器connector(十六)
本人有一段时间没有接触企业搜索引擎之连接器的开发了,连接器是涉及企业搜索引擎一个重要的组件,在数据源与企业搜索引擎中间起一个桥梁的作用,类似于数据库之JDBC,通过连接器将不同数据源的数据适配到企业搜 ...
- 企业搜索引擎开发之连接器connector(二十八)
通常一个SnapshotRepository仓库对象对应一个DocumentSnapshotRepositoryMonitor监视器对象,同时也对应一个快照存储器对象,它们的关联是通过监视器管理对象D ...
- 企业搜索引擎开发之连接器connector(二十六)
连接器通过监视器对象DocumentSnapshotRepositoryMonitor从上文提到的仓库对象SnapshotRepository(数据库仓库为DBSnapshotRepository)中 ...
- 企业搜索引擎开发之连接器connector(二十五)
下面开始具体分析连接器是怎么与连接器实例交互的,这里主要是分析连接器怎么从连接器实例获取数据的(前面文章有涉及基于http协议与连接器的xml格式的交互,连接器对连接器实例的设置都是通过配置文件操作的 ...
- 企业搜索引擎开发之连接器connector(二十四)
本人在上文中提到,连接器实现了两种事件依赖的机制 ,其一是我们手动操作连接器实例时:其二是由连接器的自动更新机制 上文中分析了连接器的自动更新机制,即定时器执行定时任务 那么,如果我们手动操作连接器实 ...
- 企业搜索引擎开发之连接器connector(二十二)
下面来分析线程执行类,线程池ThreadPool类 对该类的理解需要对java的线程池比较熟悉 该类引用了一个内部类 /** * The lazily constructed LazyThreadPo ...
- 企业搜索引擎开发之连接器connector(二十)
连接器里面衔接数据源与数据推送对象的是QueryTraverser类对象,该类实现了Traverser接口 /** * Interface presented by a Traverser. Used ...
随机推荐
- String、StringBuffer、StringBuilder的区别
在日常开发过程中String字符串估计是被用到最多的变量了,最近看了一些String.StringBuffer和StringBuilder的东西,三者都可以对字符串进行操作,他们究竟有什么区别,以及适 ...
- 谈谈Activiti中流程对象之间的关系
详细见:http://www.kafeitu.me/activiti/2012/03/22/workflow-activiti-action.html (咖啡兔好牛!) 详细见: http://blo ...
- [转载]Matlab生成Word报告
最近在进行一批来料的检验测试,一个个手动填写报告存图片太慢了,就有了种想要使用Matlab在分析完后数据可以自动生成PDF报告的想法,于是就去网上搜索了相关的资料,发现Matlab中文论坛上有xiez ...
- Android sharePreference使用
SharedPreferences类,它是一个轻量级的存储类,特别适合用于保存软件配置参数. SharedPreferences保存数据,其背后是用xml文件存放数据,文件存放在/data/data/ ...
- Call requires API level 21(Current min is 16)
Call requires API level 21(Current min is 16) Android开发中,遇到类似这种问题,如何处理? 一种办法是提升sdk最低版本到21,在Android s ...
- iOS8以后自动计算cell的高度
前提: 1.iOS系统>=8 2.cell中的每个控件布局固定,不含一些动态的模块,但是可以含有label的变化 可以采用tableView自动计算cell的高度 首先设置tableView的属 ...
- Finders Keepers
function find(arr, func) { //var num = 0; //return num; var res = arr.filter(func); if(res.length){ ...
- Java(String)
1>String类型判等 (1)请运行以下示例代码StringPool.java,查看其输出结果.如何解释这样的输出结果?从中你能总结出什么? package StringPool; publi ...
- android中ColorStateList及StateListDrawable设置Selector
写过android的代码相信大家对Selector并不陌生吧,下面来看看这段xml文件是如何定义的 <?xml version="1.0" encoding="ut ...
- bootstrap-标题
1.重置了margin-top和margin-bottom的值,h1到h3重置后margin-top的值都是20px:h4到h6重置后的值都是10px: 2.所有标题的行高都是1.1(也就是font- ...