Nutch环境搭建

1. 环境准备

HOST：Ubuntu12.04LTS

JDK： jdk-7u45-linux-i586.rpm

Nutch：apache-nutch-1.7-bin.tar.gz

Solr：solr-4.6.0-src.tgz

工作目录： /home/zephyr/

 

1.1JDK安装

为了下载快点，一下脑残下了rpm.

可Ubuntu上没有rpm工具apt-get install rpm 后rpm -ivh jdk-7u45-linux-i586.rpm

提示要使用alien

apt-get install alien alien -i jdk-7u45-linux-i586.rpm

alien -d jdk-7u45-linux-i586.rpm 生成deb文件 不可用

无奈，重新使用apt-get install oracle-java7-installer* 直接使用失败

add-apt-repository ppa:webupd8team/java

apt-get update

apt-get install oracle-java7-installer

到/etc/profile 文件的最后增加上以下几行

export JAVA_HOME=/usr/java/jdk1.7.0_45

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

export NUTCH_HOME=/home/zephyr/apache-nutch-1.7

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$NUTCH_HOME/bin:$PATH

 

1.2Tomcat安装

apt-get install tomcat7 下载tomcat

修改tomcat的端口使用情况为

vi /etc/tomcat7/server.xml

<Connector port="8088" protocol="HTTP/1.1"

connectionTimeout="20000"

URIEncoding="UTF-8"

redirectPort="8443" />

/etc/init.d/tomcat7 start

 

1.3Nutch 安装

工作目录下解压 tar xzvf apache-nutch-1.7-bin.tar.gz

测试Nutch 是否能正常工作

在urls目录touch一个seed.txt echo http://nutch.apache.org/ >> seed.txt

修改conf/regex-urlfilter.txt

   # accept anything else

     +.

为

   +^http://([a-z0-9]*\.)*nutch.apache.org/

编辑conf/nutch-site.xml

<property>

 <name>http.agent.name</name>

 <value>My Nutch Spider</value>

</property>

.nutch crawl ../urls -dir ../test -depth 1 -topN 10

报错

java/lang/NoClassDefFoundError: java/lang/Object

Exception in thread "main" java.lang.InternalError: internal error: SHA-1 not available

修正JDK

cd /usr/java/jdk1.7.0_45/lib 将pack结尾的文件都转成 jar 如：unpack200 tools.pack tools.jar

/usr/java/jdk1.7.0_45/jre/lib 也做相同操作

再操作成功。

 

1.4 Solr安装

tar xzvf solr-4.6.0-src.tgz

ivy未安装，先安装ivy, apt-get install ivy

ant compile 失败，修正ln -s -T /usr/share/java/ivy.jar /usr/share/ant/lib

/ivy.jar 编译无响应

tar xzvf solr-4.6.0.tgz

把Solr 预设的core 范例collection1 复制为core-nutch

把Nutch 提供的conf/schema-solr4.xml 覆盖掉Solr core-nutch 的conf/schema.xml

将Solr core-nutch conf/schema.xml补上一行漏掉的栏位设定<field name=”_version_” type=”long” stored=”true” indexed=”true” multiValued=”false”/>

将core.properties中的name 改为 core-nutch

nutch crawl /home/zephyr/apache-nutch-1.7/urls -solr http://192.168.153.128:8983/solr/core-nutch/ -threads 20 -depth 1 -topN 10

也可以使用nutch solrindex http://192.168.153.128:8983/solr/core-nutch/ /home/zephyr/apache-nutch-1.7/test/crawldb -linkdb /home/zephyr/apache-nutch-1.7/test/linkdb/ /home/zephyr/apache-nutc

h-1.7/test/segments/*

 

1.5 solr通过tomcat7部署

apt-get install tomcat7-admin

/etc/tomcat7编辑tomcat-users.xml

<role rolename="manager-gui"/>

<role rolename="admin-gui"/>

<user username="tomcat" password="tomcat" roles="manager-gui,admin-gui"/>

cp /home/zephyr/solr-4.6.0/example/webapps/solr.war /var/lib/tomcat7/webapps/solr.war

cp –r /home/zephyr/solr-4.6.0/example/solr /var/lib/tomcat7/solr

cd /var/lib/tomcat7/conf/Catalina/localhost

vi solr.xml

<Context docBase="/var/lib/tomcat7/webapps/solr.war" debug="0" crossContext="true" >

   <Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat7/solr" override="true" />

</Context>

cp /home/zephyr/solr-4.6.0/example/lib/ext/*.jar /usr/share/tomcat7/lib

cp /home/zephyr/solr-4.6.0/example/resources/log4j.properties /usr/share/tomcat7/lib/

vi log4j.properties 修改 solr.log=/var/logs/tomcat7/

否则solr无法在tomcat启动

“One of the things about running Solr in a container other than the embedded Jetty is that we need to setup the logging. This step is crucial, without doing this solr will never start and you will pull all of your hair out trying to figure out why”

启动运行提示出错：

SolrCore 'collection1' is not available due to init failure: Index locked for write for core collection1

cd /var/lib/tomcat7/solr/ collection1

chown tomcat7 data -R

cd /var/lib/tomcat7/solr/core-nutch

chown tomcat7 data –R

 

1.6 Nutch添加中文支持

编辑/home/zephyr/apache-nutch-1.7/conf/nutch-site.xml

<property>

<name>http.accept.language</name>

<value>ja-jp, en-us, zh-cn,en-gb,en;q=0.7,*;q=0.3</value>

<description>Value of the “Accept-Language” request header field.

   This allows selecting non-English language as default one to retrieve.

   It is a useful setting for search engines build for certain national group.

</description>

</property>