通过xpath获取对应的整个子节点信息
1.页面解析接口
package com.dajiangtai.djt_spider.service;
import com.dajiangtai.djt_spider.entity.Page;
/**
* 页面解析接口
* @author Administrator
*
*/
public interface IProcessService {
public void process(Page page);
}
2.页面解析实现类
package com.dajiangtai.djt_spider.service.impl;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.htmlcleaner.HtmlCleaner;
import org.htmlcleaner.TagNode;
import org.htmlcleaner.XPatherException;
import com.dajiangtai.djt_spider.entity.Page;
import com.dajiangtai.djt_spider.service.IProcessService;
import com.dajiangtai.djt_spider.util.HtmlUtil;
import com.dajiangtai.djt_spider.util.LoadPropertyUtil;
import com.dajiangtai.djt_spider.util.RegexUtil;
/**
* 优酷页面解析实现类
* @author Administrator
*
*/
public class YOUKUProcessService implements IProcessService{
//获取到的总播放数:16,960,789,989 其xpath为:
// /html/body/div[4]/div/div[1]/div[2]/div[2]/ul/li[11]
//这里ul最近的div[2]其class为"p-base",因此,仿写其他xpath,改成如下相对路径
private String parseAllNumber = "/body/div/div/div/div/div/ul/li[11]";
//评论数
private String parseCommentNumber = "//div[@class=\"p-base\"]/ul/li[12]";
//赞数
private String parseSupportNumber = "//div[@class=\"p-base\"]/ul/li[13]";
public void process(Page page) {
String content = page.getContent();
HtmlCleaner htmlCleaner = new HtmlCleaner();
//利用htmlCleaner对网页进行解析,得到根节点
TagNode rootNode = htmlCleaner.clean(content);
try {
Object[] evaluateXPath = rootNode.evaluateXPath(parseAllNumber);
if(evaluateXPath.length>0){
TagNode node = (TagNode)evaluateXPath[0];
System.out.println(node.getText().toString());
}
evaluateXPath = rootNode.evaluateXPath(parseCommentNumber);
if(evaluateXPath.length>0){
TagNode node = (TagNode)evaluateXPath[0];
System.out.println(node.getText().toString());
}
evaluateXPath = rootNode.evaluateXPath(parseSupportNumber);
if(evaluateXPath.length>0){
TagNode node = (TagNode)evaluateXPath[0];
System.out.println(node.getText().toString());
}
} catch (XPatherException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
3.爬虫入口类在main方法中对页面解析方法进行测试:
package com.dajiangtai.djt_spider.start;
import com.dajiangtai.djt_spider.entity.Page;
import com.dajiangtai.djt_spider.service.IDownLoadService;
import com.dajiangtai.djt_spider.service.IProcessService;
import com.dajiangtai.djt_spider.service.IStoreService;
import com.dajiangtai.djt_spider.service.impl.ConsoleStoreService;
import com.dajiangtai.djt_spider.service.impl.HttpClientDownLoadService;
import com.dajiangtai.djt_spider.service.impl.YOUKUProcessService;
/**
* 电视剧爬虫入口类
* @author Administrator
*
*/
public class StartDSJCount {
//页面下载接口
private IDownLoadService downLoadService;
private IProcessService processService;
public static void main(String[] args) {
StartDSJCount dsj = new StartDSJCount();
dsj.setDownLoadService(new HttpClientDownLoadService());
dsj.setProcessService(new YOUKUProcessService());
String url = "http://list.youku.com/show/id_z9cd2277647d311e5b692.html?spm=a2h0j.8191423.sMain.5~5~A!2.iCUyO9";
//下载页面
Page page = dsj.downloadPage(url);
//解析页面
dsj.processPage(page);
}
//下载页面方法
public Page downloadPage(String url){
return this.downLoadService.download(url);
}
//解析页面方法
public void processPage(Page page){
this.processService.process(page);
}
public IDownLoadService getDownLoadService() {
return downLoadService;
}
public void setDownLoadService(IDownLoadService downLoadService) {
this.downLoadService = downLoadService;
}
public IProcessService getProcessService() {
return processService;
}
public void setProcessService(IProcessService processService) {
this.processService = processService;
}
}
4.测试结果如下:
这样,下图中标记信息已经全部解析成功了。
通过xpath获取对应的整个子节点信息的更多相关文章
- CSS/Xpath 选择器 第几个子节点/父节点/兄弟节点
0.参考 1.初始化 In [325]: from scrapy import Selector In [326]: text=""" ...: <div> ...
- Xpath string()提取多个子节点中的文本
<div> <ul class="show"> <li>275万购昌平邻铁三居 总价20万买一居</li> <li>00 ...
- jQuery获取父级、兄弟节点的方法
一.jQuery的父节点查找方法 $(selector).parent(selector):获取父节点 $(selector).parentNode:以node[]的形式存放父节点,如果没有父节点,则 ...
- JS(基础)_总结获取页面中元素和节点的方式
一.前言 1.元素和节点的区别 2.总结获取元素的方式 3.总结获取节点的方式 二.主要内容 1.结点和元素的区别 (1)一些常见基本概念: 文档:document 元素:页面中所有的标签 结点:页面 ...
- 解析xml(当节点中有多个子节点)
概要:解析一个xml,当一个节点中又包含多个子节点如何解析,对比一个节点中不包括其他节点的情况. 一,xml样例 <cisReports batNo="查询批次号" unit ...
- xpath获取同级元素
XPath轴(XPath Axes)可定义某个相对于当前节点的节点集: 1.child 选取当前节点的所有子元素 2.parent 选取当前节点的父节点 3.descendant 选取当前节点的所有后 ...
- Dom 获取、Dom动态创建节点
一.Dom获取 1.全称:Document Object Model 文档对象模型 2.我们常用的节点类型 元素(标签)节点.文本节点.属性节点(也就是标签里的属性). 3.docum ...
- JS DOM操作(创建、遍历、获取、操作、删除节点)
创建节点 <!DOCTYPE html> <html lang="zh-CN"> <head> <meta charset="u ...
- 使用python+xpath 获取https://pypi.python.org/pypi/lxml/2.3/的下载链接
使用python+xpath 获取https://pypi.python.org/pypi/lxml/2.3/的下载链接: 使用requests获取html后,分析html中的标签发现所需要的链接在& ...
随机推荐
- HTML 学习笔记 JQuery(盒子操作)
这边博客详细的讲述一下JQuery中关于盒子模型的一些方法 offset([coordinates])方法 获取匹配元素在当前适口的相对偏移 返回的对象包含两个模型属性:top和left 以像素计.此 ...
- Android笔记之Retrofit与RxJava的组合
依赖 implementation 'com.squareup.retrofit2:retrofit:2.5.0' implementation 'com.squareup.retrofit2:con ...
- (转)nginx-rtmp-module和ffmpeg搭建实时HLS切片
1.rtmp服务器 nginx+pcre+zlib+openssl+nginx-rtmp-module ./configure \ --prefix=/usr/local/nginx \ --sbin ...
- BZOJ1833 数位DP
数位DP随便搞搞. #include<iostream> #include<cstdio> #include<cstdlib> #include<cstrin ...
- hadoop 安装配置
1.伪分布式搭建: 步骤参考: http://wenku.baidu.com/link?url=N_Sc7dqaO5HB47SmhntYZQI2tvvAjYt0mWT0fx28FDSMRYKTLUTc ...
- CodeForces - 960F Pathwalks —— 主席树(n棵线段树)
题目链接:https://vjudge.net/problem/CodeForces-960F You are given a directed graph with n nodes and m ed ...
- 51nod 1737配对
题意:给定一个n个点的带边权树, 保证n是偶数,给这个树两两配对,使得配对后的点路径和最大,输出最大值. 其实是个很简单的题,但还是被绊了.这充分说明现在连简单题都做不来了555 单独考虑每条边.每 ...
- 常用JS组件整理
1.漂亮的弹出层----artDialog http://aui.github.io/artDialog/ 2.弹出层 ------layer http://sentsin.com/jquery/la ...
- listen and translation exercise 52
Two years ago my husband bought me a bicycle. Twenty of the thirty-five years I ran my own business. ...
- babel-runtime 和 babel-polyfill
Babel 默认只转换新的 JavaScript 语法 https://excaliburhan.com/post/babel-preset-and-plugins.html babel-plugin ...