scrapy Mongodb 储存
pipelines.py
# -*- coding: utf-8 -*- # Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
import pymongo
from scrapy.exceptions import DropItem class JobsPipeline(object): # (Jobs→自己的项目名)
def process_item(self, item, spider):
return item class MongoPipeline(object):
collection_name = '表名' def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db @classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
) def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db] def close_spider(self, spider):
self.client.close() def process_item(self, item, spider):
self.db[self.collection_name].insert(dict(item))
return item
settings.py
# -*- coding: utf-8 -*- # Scrapy settings for Jobs project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://doc.scrapy.org/en/latest/topics/settings.html
# https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
# https://doc.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = 'Jobs' SPIDER_MODULES = ['Jobs.spiders']
NEWSPIDER_MODULE = 'Jobs.spiders'
DOWNLOAD_DELAY = 0
ITEM_PIPELINES = {
# 'zhaopin.pipelines.ZhaopinPipeline': 300,
'Jobs.pipelines.MongoPipeline': 400
}
MONGO_URL = 'localhost'
MONGO_DATABASE = 'shuju' # 数据库名 # Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Jobs (+http://www.yourdomain.com)' # Obey robots.txt rules
ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default)
#COOKIES_ENABLED = False # Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False # Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#} # Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'Jobs.middlewares.JobsSpiderMiddleware': 543,
#} # Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'Jobs.middlewares.JobsDownloaderMiddleware': 543,
#} # Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#} # Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
# 'Jobs.pipelines.JobsPipeline': 300,
#} # Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' # 设置日志级别为Error,低于次级别的日志就出不来
LOG_LEVEL = 'ERROR' # 禁用日志输出,所有日志都出不来
# LOG_ENABLED=False # 保持中文,不被Unicode编码
FEED_EXPORT_ENCODING = 'utf-8'
scrapy Mongodb 储存的更多相关文章
- 用 mongodb 储存多态消息/提醒类数据(转)
原文:http://codecampo.com/topics/66 前天看到 javaeye 计划采用mongoDB实现网站全站消息系统,很有同感,mongodb 很适合储存消息类数据.之前讨论了如何 ...
- python scrapy+Mongodb爬取蜻蜓FM,酷我及懒人听书
1.初衷:想在网上批量下载点听书.脱口秀之类,资源匮乏,大家可以一试 2.技术:wireshark scrapy jsonMonogoDB 3.思路:wireshark分析移动APP返回的各种连接分类 ...
- scrapy+mongodb
我们都知道scrapy适合爬取大量的网站信息,爬取到的信息储存到数据库显然需要更高的效率,scrapy配合mongodb是非常合适的,这里记录一下如何在scrapy中配置mongodb. 文件结构 $ ...
- scrapy+mongodb报错 TypeError: name must be an instance of str
经过各种排查,最后找到原因,在settings文件中配置文件大小写写错了,在pipelines中 mongo_db=crawler.settings.get('MONGODB_DB'),get 获取的 ...
- 利用Scrapy爬取所有知乎用户详细信息并存至MongoDB
欢迎大家关注腾讯云技术社区-博客园官方主页,我们将持续在博客园为大家推荐技术精品文章哦~ 作者 :崔庆才 本节分享一下爬取知乎用户所有用户信息的 Scrapy 爬虫实战. 本节目标 本节要实现的内容有 ...
- PHP操作MongoDB简明教程(转)
转自:http://blog.sina.com.cn/s/blog_6324c2380100ux2m.html MongoDB是最近比较流行的NoSQL数据库,网络上关于PHP操作MongoDB的资料 ...
- MongoDB与PHP的添加、修改、查询、删除
链接数据库使用下面的代码创建一个数据库链接 <?php$connection = new Mongo(); //链接到 localhost:27017$connection = new Mong ...
- 爬虫框架之Scrapy(一)
scrapy简介 scrapy是一个用python实现为了爬取网站数据,提取结构性数据而编写的应用框架,功能非常的强大. scrapy常应用在包括数据挖掘,信息处理或者储存历史数据的一系列程序中. s ...
- mongodb shell和Node.js driver使用基础
开始: Mongo Shell 安装后,输入mongo进入控制台: //所有帮助 > help //数据库的方法 > db.help() > db.stats() //当前数据库的状 ...
随机推荐
- (最完美)红米手机4的USB调试模式在哪里开启的经验
每次我们使用安卓手机通过数据线连接上Pc的时候,或者使用的有些应用比如我们团队营销部门每次使用的应用引号精灵,之前的老版本就需要开启Usb开发者调试模式下使用,现每次新版本不需要了,如果手机没有开启U ...
- TestNG+Java+Selenium+Maven 代码实例
搭环境时注意不要忘记testng的jar包,selenium的jar包 package com.guge.test; import org.testng.annotations.Test;import ...
- 正式表达式判断私有 IP 地址
正式表达式判断私有 IP 地址 ^1(((0|27)(.(([1-9]?|1[0-9])[0-9]|2([0-4][0-9]|5[0-5])))|(72.(1[6-9]|2[0-9]|3[01]) ...
- 腾讯、爱奇艺、优酷等vip视频在线解析
http://vip.mist.xin 菠萝蜜TV全网VIP视频在线看 菠萝蜜tv http://jx.mist.xin 全网VIP视频在线解析接口 免费全网影视VIP视频vip会员免广告看电影!亲们 ...
- python-day5内置模块time、range、sys、os、shelve、xml、max等
@os树状目录 import os,os.path def showdir(path,depth): if depth==0: print(path) for item in ...
- 1. maven 手动添加jar包
mvn install:install-file -Dfile=isc_sso_agent-1.0.jar -DgroupId=com.sgcc.isc -DartifactId=isc_sso_ag ...
- mockito 异常Reason: java.io.IOException: invalid constant type: 18
原因: mockito内部使用的javassit的版本不一致导致的,修改为一直版本即可. 异常内容: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jd ...
- Docker技术应用场景(转载)
场景一:节省项目环境部署时间 1.单项目打包 每次部署项目到测试.生产等环境,都要部署一大堆依赖的软件.工具,而且部署期间出现问题几率很大,不经意就花费了很长时间. Docker主要理念就是环境打包部 ...
- thinkphp5.1 退出登陆操作
使用Session:: 静态方法即可
- cmd命令行模式开启或关闭 windows功能 (转载)
1,用管理员身份运行cmd. 2,运行 dism /online /Get-Features >C:\1.txt,这样会在C盘生成一个txt文件,文件内容是功能列表,可以对照“打开或关闭win ...