Setting up Nutch 2.1 with MySQL to handle UTF-8
原文地址: http://nlp.solutions.asia/?p=180
These instructions assume Ubuntu 12.04 and Java 6 or 7 installed and JAVA_HOME configured.
Install MySQL Server and MySQL Client using the Ubuntu software center or sudo apt-get install mysql-server mysql-client
at the command line.
As MySQL defaults to latin (are we still in the 1990s?) we need to edit sudo vi /etc/mysql/my.cnf
and under [mysqld] add
innodb_file_format=barracuda
innodb_file_per_table=true
innodb_large_prefix=true
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
max_allowed_packet=500M
The innodb options are to help deal with the small primary key size restriction of MySQL. Restart your machine for the changes to take effect. The max_allowed_packet option is so you don’t run into issues as your database and the pages you store in it get larger.
Check to make sure MySQL is running by typing sudo netstat -tap | grep mysql
and you should see something like
tcp 0 0 localhost:mysql *:* LISTEN
We need to set up the nutch database manually as the current Nutch/Gora/MySQL generated db schema defaults to latin. Log into mysql at the command line using your previously set up MySQL id and password type
mysql -u xxxxx -p
then in the MySQL editor type the following:
CREATE DATABASE nutch DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
and enter followed by
use nutch;
and enter and then copy and paste the following altogether:
CREATE TABLE `webpage` (
`id` varchar(767) NOT NULL,
`headers` blob,
`text` mediumtext DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`markers` blob,
`parseStatus` blob,
`modifiedTime` bigint(20) DEFAULT NULL,
`score` float DEFAULT NULL,
`typ` varchar(32) CHARACTER SET latin1 DEFAULT NULL,
`baseUrl` varchar(767) DEFAULT NULL,
`content` longblob,
`title` varchar(2048) DEFAULT NULL,
`reprUrl` varchar(767) DEFAULT NULL,
`fetchInterval` int(11) DEFAULT NULL,
`prevFetchTime` bigint(20) DEFAULT NULL,
`inlinks` mediumblob,
`prevSignature` blob,
`outlinks` mediumblob,
`fetchTime` bigint(20) DEFAULT NULL,
`retriesSinceFetch` int(11) DEFAULT NULL,
`protocolStatus` blob,
`signature` blob,
`metadata` blob,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
ROW_FORMAT=COMPRESSED
DEFAULT CHARSET=utf8mb4;
Then type enter. You are done setting up the MySQL database for Nutch.
Set up Nutch 2.1 by downloading the latest version from http://www.apache.org/dyn/closer.cgi/nutch/. Untar the contents of the file you just downloaded and going forward we will refer to this folder as ${APACHE_NUTCH_HOME}.
From inside the nutch folder ensure the MySQL dependency for Nutch is available by editing the following in ${APACHE_NUTCH_HOME}/ivy/ivy.xml
<!– Uncomment this to use MySQL as database with SQL as Gora store. –>
<dependency org=”mysql” name=”mysql-connector-java” rev=”5.1.18″ conf=”*->default”/>
Edit the ${APACHE_NUTCH_HOME}/conf/gora.properties file either deleting or commenting out the Default SqlStore Properties using #. Then add the MySQL properties below replacing xxxxx with the user and password you set up when installing MySQL earlier.
###############################
# MySQL properties #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
gora.sqlstore.jdbc.user=xxxxx
gora.sqlstore.jdbc.password=xxxxx
Edit the ${APACHE_NUTCH_HOME}/conf/gora-sql-mapping.xml file changing the length of the primarykey from 512 to 767 in both places.
<primarykey column=”id” length=”767″/>
Configure ${APACHE_NUTCH_HOME}/conf/nutch-site.xml to put in a name in the value field under http.agent.name. It can be anything but cannot be left blank. Add additional languages if you want (I have added Japanese ja-jp below) and utf-8 as default as well. You must specify Sqlstore.
<property>
<name>http.agent.name</name>
<value>Your Nutch Spider</value>
</property>
<property>
<name>http.accept.language</name>
<value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept-Language” request header field.
This allows selecting non-English language as default one to retrieve.
It is a useful setting for search engines build for certain national group.
</description>
</property>
<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>The character encoding to fall back to when no other information
is available</description>
</property>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>The Gora DataStore class for storing and retrieving data.
Currently the following stores are available: ….
</description>
</property>
Install ant using the Ubuntu software center or sudo apt-get install ant
at the command line.
From the command line cd
to your nutch folder type ant runtime
This may take a few minutes to compile.
Start your first crawl by typing the lines below at the terminal (replace ‘http://nutch.apache.org/’ with whatever site you want to crawl):
cd ${APACHE_NUTCH_HOME}/runtime/local
mkdir -p urls
echo 'http://nutch.apache.org/' > urls/seed.txt
bin/nutch crawl urls -depth 3 -topN 5
You can easily add more urls to search by hand in seed.txt if you want. For the crawl, depth is the number of rounds of generate/fetch/parse/update you want to do (not depth of links as you might think at first) and topNis the max number of links you want to actually parse each time. Note however Nutch keeps track of all links it encounters in the webpage table (it just limits the amount it actually parses to TopN so don’t be surprised by seeing many more rows in the webpage table than you expect by limiting with TopN).
Check your crawl results by looking at the webpage table in the nutch database.
mysql -u xxxxx -p
use nutch;
SELECT * FROM nutch.webpage;
You should see the results of your crawl (around 159 rows). It will be hard to read the columns so you may want to install MySQL Workbench via sudo apt-get install mysql-workbench
and use that instead for viewing the data. You may also want to run the following SQL command select * from webpage where status = 2;
to limit the rows in the webpage table to only urls that were actually parsed.
Set up and index with Solr If you are using Nutch 2.1 at this time you are into the bleeding edge and probably want the latest version of Solr 4.0 as well. Untar it to to $HOME/apache-solr-4.0.0-XXXX. This folder will be now referred to as ${APACHE_SOLR_HOME}.
Download http://nlp.solutions.asia/wp-content/uploads/2012/08/schema.xml and use it to replace ${APACHE_SOLR_HOME}/example/solr/collection1/conf/schema.xml.
From the terminal start solr:cd ${APACHE_SOLR_HOME}/example
java -jar start.jar
You can check this is running by opening http://localhost:8983/solr in your web browser.
Leave that terminal running and from a different terminal type the following:cd ${APACHE_NUTCH_HOME}/runtime/local/
bin/nutch solrindex http://127.0.0.1:8983/solr/ -reindex
You can now run queries using Solr versus your crawled content. Openhttp://localhost:8983/solr/#/collection1/query and assuming you have crawled nutch.apache.org in the input box titled “q” you can do a search by inputting text:nutch and you should see something like this:
There remains a lot to configure to get a good web search going but you are at least started.
Setting up Nutch 2.1 with MySQL to handle UTF-8的更多相关文章
- 安装mysql sever 向导失败,最后一步无响应
在配置apache+php+mysql环境的时候,apache和php都可以运行,这里提供两个安装教程(window环境) http://apps.hi.baidu.com/share/detail/ ...
- 浅谈mysql的两阶段提交协议
前两天和百度的一个同学聊MySQL两阶段提交,当时自信满满的说了一堆,后来发现还是有些问题的理解还是比较模糊,可能是因为时间太久了,忘记了吧.这里再补一下:) 5.3.1事务提交流程 MySQL的事务 ...
- MySQL5中大数据错误:Packet for query is too large (****** > ******). You can change this value on the server by setting the max_allowed_packet' variable.;
使用的MySQL数据库版本:5.5 插入或更新字段有大数据时(大于1M),会出现如下错误: ### Cause: com.mysql.jdbc.PacketTooBigException: Packe ...
- mysql start server faild
可能没卸载干净...在安装mysql数据库时,如果重新安装,很容易遇见apply security setting error,即在配置mysql启动服务时,在启动apply security set ...
- mysql安装过程中出现的错误问题解决方案
最近在学Django,因为与数据库相关,所以我下载并安装了MySQL,安装的过程真的是一把辛酸泪啊.安装过后,查看是否可以使用,出现了cann't connect to mysql server这个错 ...
- mysql的两阶段提交协议
http://www.cnblogs.com/hustcat/p/3577584.html 前两天和百度的一个同学聊MySQL两阶段提交,当时自信满满的说了一堆,后来发现还是有些问题的理解还是比较 ...
- 初次配置eclipse, jdk, tomcat, maven, mysql, alt+/
eclipse 官网下载eclipse-inst-win64.exe, 选择安装java ee. jdk 官网下载jdk-8u102-windows-x64.exe, next到底. 接下来配置环境变 ...
- MySQL 5.7 在windows下修改max_allowed_packet变量
(一)执行sql遇到的错误如下: ### Cause: com.mysql.jdbc.PacketTooBigException: Packet for query is too large (387 ...
- django中sqlite迁移mysql
sqlite数据迁移 1 数据备份 django中打开terminalpython manage.py dumpdata authorization > authorization_data.j ...
随机推荐
- Android无法调用JS的问题解决
1.启用JS webView.getSettings().setJavaScriptEnabled(true); 2.需要使用WebChromeClient(如果没有这个Client,很多东西不会响应 ...
- java jxl 向Excel中追加数据而不覆盖原来数据的例子
向先原来就有数据的Excel写数据是不会覆盖原有的数据,只是在追加数据. public class Excel { public Excel() { } public void ...
- [C#搜片神器] 之P2P中DHT网络爬虫原理
继续接着上一篇写:使用C#实现DHT磁力搜索的BT种子后端管理程序+数据库设计(开源)[搜片神器] 昨天由于开源的时候没有注意运行环境,直接没有考虑下载BT种子文件时生成子文件夹,可能导致有的朋友运行 ...
- angularJS+requireJS并集成karma测试实践
最近在为下一个项目做前端技术选型,Angular是必须要用的(BOSS指定,个人感觉也不错,开发效率会很高).由于需要加载的JS很多,所以打算看看angular和requirejs一起用会怎么样.在g ...
- Linux基本命令(10)其他命令
其他命令 命令 功能 命令 功能 echo 显示一字串 passwd 修改密码 clear 清除显示器 lpr 打印 lpq 查看在打印队列中等待的作业 lprm 取消打印队列中的作业 10.1 ec ...
- 【LeetCode】88 - Merge Sorted Array
Given two sorted integer arrays nums1 and nums2, merge nums2 into nums1 as one sorted array. Note:Yo ...
- [Asp.net MVC]Asp.net MVC5系列——Razor语法
Razor视图引擎是Asp.net MVC3中新扩展的内容,并且也是它的默认视图引擎.还有另外一种Web Forms视图引擎.通过前面的文章可知在Asp.net mvc5中创建视图,默认使用的是Raz ...
- log4net--帮助程序员将日志信息输出到各种目标(控制台、文件、数据库等)的工具
1. log4net库是Apache log4j框架在Microsoft .NET平台的实现,是一个帮助程序员将日志信息输出到各种目标(控制台.文件.数据库等)的工具. 2. Log4net的结构如下 ...
- Best Practices for Speeding Up Your Web Site
The Exceptional Performance team has identified a number of best practices for making web pages fast ...
- LeetCode(4) - Median of Two Sorted Arrays
题目要求很简单,就是给你两个已经排好序的数组nums1(长度为m)和nums2(长度为n),找出他们的中间值.返回值类型double是因为如果数字个数是偶数个,就要返回中间两个数的平均值.这题最简单的 ...