scrapy的splash 的简单使用

安装Splash（拉取镜像下来）
docker pull scrapinghub/splash
安装scrapy-splash
pip install scrapy-splash
启动容器
docker run -p 8050:8050 scrapinghub/splash
setting 里面配置
SPLASH_URL = 'http://192.168.99.100:8050' #（很重要写错了会出目标电脑积极拒绝）
添加Splash中间件，指定优先级
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
设置Splash自己的去重过滤器
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
缓存后台存储介质
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' # 以上两条必加
eg:
import scrapy
from scrapy_splash import SplashRequest
class JsSpider(scrapy.Spider):
name = "jd"
allowed_domains = ["jd.com"]
start_urls = [
"http://www.jd.com/"
]
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, args={'wait': 0.5})
def parse(self, response):
print('----------使用splash爬取京东网首页异步加载内容-----------')
rs=response.xpath('//span[@class="ui-areamini-text"]/text()').extract()[0]
print(rs)
print('---------------success----------------')
官方文档:https://pypi.python.org/pypi/scrapy-splash