WEB数据挖掘(十六)——Aperture数据抽取(9):数据源
One of the central concepts of Aperture is the notion of a DataSource. A DataSource contains all information necessary to locate the individual information resources in a physical source. For example, a FileSystemDataSource holds a root directory, a set of patterns that describe what files to include or exclude, a maximum depth, etc., thereby effectively describing a set of files.
One of the main purposes of a DataSource is to hold all data needed by a Crawlers
to crawl the physical source and retrieve all the individual resources
in it. There are quite a few DataSource subclasses in Aperture. The
following diagram contains a selection of them.
The specific DataSource implementations available at the moment contain
specific 'get' and 'set' methods for the configuration properties
accepted by the data source. Thus providing a convenient interface and
abstracting from the underlying RDF properties. All configuration data
is stored in a RDFContainer. Each data source type comes with it's own
specific properties. There is also a set of generic properties used by
many data source types (username, password etc.). You can have a look at
the source code of the DataSource implementation class of your choosing
to see which properties are used. Note that the data source classes are
not stored in the SVN. They are generated automatically from an RDF
file with the description of the class. (like this one). The classes are generated by a maven plugin, by adding appropriate entries in the datasource module pom.xml file similar to these.
If you'd like to develop your own data source implementation, try to
mimic the existing implementations or ask at the aperture-devel for
help.
It is worth mentioning, that DataSource classes only DESCRIBE a data
source. They don't contain any resources that would enable direct access
to the source (such as InputStreams, or Readers, whatever...). (At
least it was not the intention of the designers). Any such resource is
obtained by the crawler at the start of crawl and may be encapsulated in
a DataObject returned by an Accessor or crawler. The following code
demonstrates how to create and configure a FileSystemDataSource:
// determine the root folder of the source
File rootFolder = new File("D:\\path\\to\\the\\root\\folder");
// create the model that will store the data source configure
Model model = RDF2Go.getModelFactory().createModel();
// don't forget to open it before use
model.open();
// determine a URI to identify the DataSource
URI id = model.createURI("urn:test:testsource");
// wrap the model in an RDFContainer
RDFContainer configuration = new RDFContainerImpl(model,id);
// create the DataSource instance
FileSystemDataSource source = new FileSystemDataSource();
// set the configuration (it is empty at the moment)
source.setConfiguration(configuration)
// and set the rootFolder (you can do it now)
source.setRootFolder(rootFolder.getAbsolutePath());
WEB数据挖掘(十六)——Aperture数据抽取(9):数据源的更多相关文章
- WCF技术剖析之十六:数据契约的等效性和版本控制
原文:WCF技术剖析之十六:数据契约的等效性和版本控制 数据契约是对用于交换的数据结构的描述,是数据序列化和反序列化的依据.在一个WCF应用中,客户端和服务端必须通过等效的数据契约方能进行有效的数据交 ...
- Spring MVC 使用介绍(十六)数据验证 (三)分组、自定义、跨参数、其他
一.概述 除了依赖注入.方法参数,Bean Validation 1.1定义的功能还包括: 1.分组验证 2.自定义验证规则 3.类级别验证 4.跨参数验证 5.组合多个验证注解 6.其他 二.分组验 ...
- java web 学习十六(JSP指令)
一.JSP指令简介 JSP指令(directive)是为JSP引擎而设计的,它们并不直接产生任何可见输出,而只是告诉引擎如何处理JSP页面中的其余部分. 在JSP 2.0规范中共定义了三个指令: pa ...
- .NET面试题系列(十六)数据库面试题
数据库事务的四大特性 原子性(Atomicity) 原子性是指事务包含的所有操作要么全部成功,要么全部失败回滚.因此事务的操作如果成功就必须要完全应用到数据库,如果操作失败则不能对数据库有任何影响. ...
- MapReduce(十六): 写数据到HDFS的源代码分析
1) LineRecordWriter负责把Key,Value的形式把数据写入到DFSOutputStream watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZ ...
- WCF技术剖析之十五:数据契约代理(DataContractSurrogate)在序列化中的作用
原文:WCF技术剖析之十五:数据契约代理(DataContractSurrogate)在序列化中的作用 [爱心链接:拯救一个25岁身患急性白血病的女孩[内有苏州电视台经济频道<天天山海经> ...
- Kali Linux Web 渗透测试视频教程— 第十六课-拒绝服务攻击
Kali Linux Web 渗透测试视频教程— 第十六课-拒绝服务攻击 文/玄魂 目录 Kali Linux Web 渗透测试视频教程— 第十六课-拒绝服务攻击................... ...
- ASP.NET Core 2.2 : 十六.扒一扒新的Endpoint路由方案 try.dot.net 的正确使用姿势 .Net NPOI 根据excel模板导出excel、直接生成excel .Net NPOI 上传excel文件、提交后台获取excel里的数据
ASP.NET Core 2.2 : 十六.扒一扒新的Endpoint路由方案 ASP.NET Core 从2.2版本开始,采用了一个新的名为Endpoint的路由方案,与原来的方案在使用上差别不 ...
- 第三百三十六节,web爬虫讲解2—urllib库中使用xpath表达式—BeautifulSoup基础
第三百三十六节,web爬虫讲解2—urllib库中使用xpath表达式—BeautifulSoup基础 在urllib中,我们一样可以使用xpath表达式进行信息提取,此时,你需要首先安装lxml模块 ...
随机推荐
- BZOJ 3295 动态逆序对
调了好久.... 转化成三维偏序,cdq处理. 好像比较快? #include<iostream> #include<cstdio> #include<cstring&g ...
- BZOJ 1010 玩具装箱
斜率优化. 事实上是选一个大于某个数的最小斜率.维护下凸壳. #include<iostream> #include<cstdio> #include<cstring&g ...
- 【英语】Bingo口语笔记(55) - work系列
- 【基础数学】质数,约数,分解质因数,GCD,LCM
1.质数: 质数(prime number)又称素数,有无限个.一个大于1的自然数,除了1和它本身外,不能整除以其他自然数(质数),换句话说就是该数除了1和它本身以外不再有其他的因数. 2.约数: 如 ...
- 安卓 Pickers(选择器)
概述 安卓提供了现成的对话框,让用户选择一个时间或日期.每一个选择器控制时间(小时,分钟,AM/PM)或日期(月,日,年)的每一部分的选择.使用这些选择器帮助 确保用户正确的,格式化的,和适合的选择一 ...
- xxx
<div style="position:absolute;left:0px;top:50px;width:1300px;height:550px;"><d ...
- Apache加载PHP.ini顺序
网上找到一份关于Apache加载PHP.ini顺序的文档: (1) apache的httpd.conf中的PhpIniDir: (2) 注册表键值:HKEY_LOCAL_MACHINE->SOF ...
- 深入理解JavaScript闭包(closure)
最近在网上查阅了不少javascript闭包(closure)相关的资料,写的大多是非常的学术和专业.对于初学者来说别说理解闭包了,就连文字叙述都很难看懂.撰写此文的目的就是用最通俗的文字揭开Java ...
- WPF的控件Binding的ElementName/RelativeSource具体用法
<TextBlock Name="_txtSickBedNo" FontStyle="Normal" Foreground="Black&quo ...
- crontab的安装及crontab命令介绍
前一天学习了 at 命令是针对仅运行一次的任务,循环运行的例行性计划任务,linux系统则是由 cron (crond) 这个系统服务来控制的.Linux 系统上面原本就有非常多的计划性工作,因此这个 ...