Nutch的配置（使用MySQL作为数据存储）

首先先从http://www.apache.org/dyn/closer.cgi/nutch/下载安装包

这里假定nutch的根目录为：${APACHE_NUTCH_HOME}

配置${APACHE_NUTCH_HOME}/ivy/ivy.xml，确保Nutch使用MySQL作为数据存储

将

1. <dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>

<dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>

改成

1. <dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>

<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>

取消以下行的注释

1. <dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />

<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />

取消以下的行注释使用Mysql作为gora存储

1. <!-- Uncomment this to use MySQL as database with SQL as Gora store. -->
2. <dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>

<!-- Uncomment this to use MySQL as database with SQL as Gora store. -->
<dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>

编辑${APACHE_NUTCH_HOME}/conf/gora.properties

添加以下代码激活MySQL的配置

1. ###############################
2. # MySQL properties #
3. ###############################
4. gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
5. gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
6. gora.sqlstore.jdbc.user=xxxxx
7. gora.sqlstore.jdbc.password=xxxxx

###############################
# MySQL properties #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
gora.sqlstore.jdbc.user=xxxxx
gora.sqlstore.jdbc.password=xxxxx

编辑 ${APACHE_NUTCH_HOME}/conf/gora-sql-mapping.xml ，将主键的长度由512改成767

1. <primarykey column="id" length="767"/>

<primarykey column="id" length="767"/>

配置${APACHE_NUTCH_HOME}/conf/nutch-site.xml

在 http.agent.name字段下增加一个名字，可以是任意值但不能为空！  如果需要的话可以添加额外的语言（例如en为英语），同时也可以设置默认编码格式为utf-8

1. <property>
2. <name>http.agent.name</name>
3. <value>YourNutchSpider</value>
4. </property>
5. <property>
6. <name>http.accept.language</name>
7. <value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>
8. <description>Value of the "Accept-Language" request header field.
9. This allows selecting non-English language as default one to retrieve.
10. It is a useful setting for search engines build for certain national group.
11. </description>
12. </property>
13. <property>
14. <name>parser.character.encoding.default</name>
15. <value>utf-8</value>
16. <description>The character encoding to fall back to when no other information
17. is available</description>
18. </property>
19. <property>
20. <name>storage.data.store.class</name>
21. <value>org.apache.gora.sql.store.SqlStore</value>
22. <description>The Gora DataStore class for storing and retrieving data.
23. Currently the following stores are available: ....
24. </description>
25. </property>

<property>
<name>http.agent.name</name>
<value>YourNutchSpider</value>
</property>

<property>

<name>http.accept.language</name>

<value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>

<description>Value of the "Accept-Language" request header field.

This allows selecting non-English language as default one to retrieve.

It is a useful setting for search engines build for certain national group.

</description>

</property>

<property>

<name>parser.character.encoding.default</name>

<value>utf-8</value>

<description>The character encoding to fall back to when no other information

is available</description>

</property>

<property>

<name>storage.data.store.class</name>

<value>org.apache.gora.sql.store.SqlStore</value>

<description>The Gora DataStore class for storing and retrieving data.

Currently the following stores are available: ....

</description>

</property>

在命令行下输入 sudo

apt-get install ant 安装配置ant

在命令行界面使用cd切换到nutch的根目录

可以在终端中输入以下指令开始你的第一个爬虫工作





1. cd ${APACHE_NUTCH_HOME}/runtime/local
2. mkdir -p urls
3. echo 'http://nutch.apache.org/' > urls/seed.txt

cd ${APACHE_NUTCH_HOME}/runtime/local
mkdir -p urls
echo 'http://nutch.apache.org/' > urls/seed.txt

Nutch 2.2使用以下命令开始爬虫，设置线程数为30

1. bin/nutch crawl urls -threads 30

bin/nutch crawl urls -threads 30

要查看爬取的数据时，进入数据库中输入以下指令即可查看



mysql -u xxxxx -p

use nutch;

SELECT * FROM nutch.webpage;

翻译源：http://wiki.apache.org/nutch/#Nutch_2.X_tutorial.28s.29