Nutch2.1+mysql+solr3.6.1+中文网站抓取

1、mysql 数据库配置

linux mysql安装步骤省略。

在首先进入/etc/my.cnf （mysql为5.1的话就不用修改my.cnf，会导致mysql不能启动）
在[mysqld]

下添加：

innodb_file_format=barracuda

innodb_file_per_table=true

innodb_large_prefix=true

character-set-server=utf8mb4

collation-server=utf8mb4_unicode_ci

创建表：

CREATE DATABASE nutch5 DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
use nutch5;
CREATE TABLE `webpage` (

        `id` varchar() CHARACTER SET latin1 NOT NULL,

        `headers` blob,

        `text` mediumtext DEFAULT NULL,

        `status` int() DEFAULT NULL,

        `markers` blob,

        `parseStatus` blob,

        `modifiedTime` bigint() DEFAULT NULL,

        `score` float DEFAULT NULL,

        `typ` varchar() CHARACTER SET latin1 DEFAULT NULL,

        `baseUrl` varchar() CHARACTER SET latin1 DEFAULT NULL,

        `content` mediumblob,

        `title` varchar() DEFAULT NULL,

        `reprUrl` varchar() CHARACTER SET latin1 DEFAULT NULL,

        `fetchInterval` int() DEFAULT NULL,

        `prevFetchTime` bigint() DEFAULT NULL,

        `inlinks` mediumblob,

        `prevSignature` blob,

        `outlinks` mediumblob,

        `fetchTime` bigint() DEFAULT NULL,

        `retriesSinceFetch` int() DEFAULT NULL,

        `protocolStatus` blob,

        `signature` blob,

        `metadata` blob,

        PRIMARY KEY (`id`)

    ) ENGINE=InnoDB DEFAULT CHARSET=utf8; 

或者用数据库备份sql：

先创建数据库：

CREATE DATABASE nutch2 DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

-- MySQL dump 10.13  Distrib 5.6., for Win64 (x86_64)
--
-- Host: yqxt    Database: nutch
-- ------------------------------------------------------
-- Server version    5.6.

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Table structure for table `webpage`
--

DROP TABLE IF EXISTS `webpage`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `webpage` (
  `id` varchar() CHARACTER SET latin1 NOT NULL,
  `headers` blob,
  `text` mediumtext,
  `status` int() DEFAULT NULL,
  `markers` blob,
  `parseStatus` blob,
  `modifiedTime` bigint() DEFAULT NULL,
  `score` float DEFAULT NULL,
  `typ` varchar() CHARACTER SET latin1 DEFAULT NULL,
  `baseUrl` varchar() CHARACTER SET latin1 DEFAULT NULL,
  `content` mediumblob,
  `title` varchar() DEFAULT NULL,
  `reprUrl` varchar() CHARACTER SET latin1 DEFAULT NULL,
  `fetchInterval` int() DEFAULT NULL,
  `prevFetchTime` bigint() DEFAULT NULL,
  `inlinks` mediumblob,
  `prevSignature` blob,
  `outlinks` mediumblob,
  `fetchTime` bigint() DEFAULT NULL,
  `retriesSinceFetch` int() DEFAULT NULL,
  `protocolStatus` blob,
  `signature` blob,
  `metadata` blob,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

-- Dump completed on --  ::

2、安装nutch2.1
A、 nutch下载地址：http://apache.etoak.com/nutch/2.1/apache-nutch-2.1-src.zip

下载完成后家压缩， 

B、以下将nutch的根目录定位${APACHE_NUTCH_HOME}.

C、配置nutch对mysql的支持，修改${APACHE_NUTCH_HOME}/ivy/ivy.xml文件

将这行的注释取消default”/<

修改${APACHE_NUTCH_HOME}/conf/gora.properties文件，

注释默认存储配置

[html]view plaincopyprint?

1. ###############################
2. # Default SqlStore properties #
3. ###############################
4. #gora.sqlstore.jdbc.driver=org.hsqldb.jdbc.JDBCDriver
5. #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest
6. #gora.sqlstore.jdbc.user=sa
7. #gora.sqlstore.jdbc.password=
8. 取消以下代码注释，
9. ###############################
10. # MySQL properties
11. ################################
12. gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
13. gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
14. gora.sqlstore.jdbc.user=xxxxx（mysql用户名）
15. gora.sqlstore.jdbc.password=xxxxx（mysql密码）

 

D、修改${APACHE_NUTCH_HOME}/conf/nutch-site.xml 加入如下代码：

[html]view plaincopyprint?

1. < span>property<
2. < span>name<http.agent.namename<
3. < span>value<Your Nutch Spidervalue<
4. property<
5. < span>property<
6. < span>name<http.accept.languagename<
7. < span>value<ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3value<
8. < span>description<Value of the “Accept-Language” request header field.
9. This allows selecting non-English language as default one to retrieve.
10. It is a useful setting for search engines build for certain national group.
11. description<
12. property<
13. < span>property<
14. < span>name<parser.character.encoding.defaultname<
15. < span>value<utf-8value<
16. < span>description<The character encoding to fall back to when no other information
17. is availabledescription<
18. property<
19. < span>property<
20. < span>name<storage.data.store.classname<
21. < span>value<org.apache.gora.sql.store.SqlStorevalue<
22. < span>description<The Gora DataStore class for storing and retrieving data.
23. Currently the following stores are available: ….
24. description<
25. property<

E、使用ant编译 ${APACHE_NUTCH_HOME} 。

 

F、设置待抓取的网站

cd ${APACHE_NUTCH_HOME}/runtime/local 
mkdir -p urls 
echo 'http://nutch.apache.org/' < urls/seed.txt

G、执行爬行操作： bin/nutch crawl urls -depth 3 -topN 5

执行完在mysql中即可以查看到爬虫抓取的内容

 

3、安装solr，对nutch抓取的内容进行索引

（注意：参考资料中推荐使用solr4.0版本，4.0的两个版本我都试了，没有成功，所以替换为3.6.1版本） 

solr下载地址：http://www.fayea.com/apache-mirror/lucene/solr/3.6.1/apache-solr-3.6.1.zip

 

A、解压缩下载包，

B、下载 http://nlp.solutions.asia/wp-content/uploads/2012/08/schema.xml替换${APACHE_SOLR_HOME}/example/solr/conf/schema.xml.

C、启动solr

cd ${APACHE_SOLR_HOME}/example

java -jar start.jar

D、在浏览器输入地址http://localhost:8983/solr 测试是否启动成功。

E、另起linux终端，输入如下命令，使solr对nutch抓取内容进行索引。

cd ${APACHE_NUTCH_HOME}/runtime/local/

bin/nutch solrindex http://127.0.0.1:8983/solr/ -reindex

4、测试

在浏览器输入 http://localhost:8983/solr ，看到如下界面：

在文本框中输入 content:nutch 点击查询，便可查处内容中包含nutch关键字的网页