FIRST SCRAPY PRJ

zpc@Lenovo-PC:/prj/pyscrapy/a$ scrapy startproject helloword
New Scrapy project 'helloword' created in:
    /cygdrive/e/01.prj/pyscrapy/a/helloword

You can start your first spider with:
    cd helloword
    scrapy genspider example example.com

 

zpc@Lenovo-PC:/prj/pyscrapy/a/helloword$ scrapy genspider baidu www.baidu.com
Created spider 'baidu' using template 'basic' in module:
  helloword.spiders.baidu

 

问题:

---

zpc@Lenovo-PC:/prj/pyscrapy/a/tutorial$ scrapy crawl dmoz
/cygdrive/e/01.prj/pyscrapy/a/tutorial/tutorial/spiders/dmoz_spider.py:3: ScrapyDeprecationWarning: tutorial.spiders.dmoz_spider.DmozSpider inherits from deprecated class scrapy.spider.BaseSpider, please inherit from scrapy.spider.Spider. (warning only on first subclass, there may be others)
  class DmozSpider(BaseSpider):
2014-12-17 11:32:38+0000 [scrapy] INFO: Scrapy 0.24.4 started (bot: tutorial)
2014-12-17 11:32:38+0000 [scrapy] INFO: Optional features available: ssl, http11
2014-12-17 11:32:38+0000 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'}
2014-12-17 11:32:40+0000 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-12-17 11:32:41+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-12-17 11:32:41+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-12-17 11:32:41+0000 [scrapy] INFO: Enabled item pipelines:
2014-12-17 11:32:41+0000 [dmoz] INFO: Spider opened
2014-12-17 11:32:41+0000 [dmoz] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-12-17 11:32:41+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2014-12-17 11:32:41+0000 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080

---

 

iconv [选项...] [文件...]
有如下选项可用:
输入/输出格式规范：
-f, --from-code=名称 原始文本编码
-t, --to-code=名称 输出编码
信息：
-l, --list 列举所有已知的字符集
输出控制：
-c 从输出中忽略无效的字符
-o, --output=FILE 输出文件
-s, --silent 关闭警告
--verbose 打印进度信息

iconv -f utf-8 -t gb2312 /server_test/reports/software_.txt > /server_test/reports/software_asserts.txt

 

URL encode decode

http://kb.cnblogs.com/page/133765/

 

(1) url编码：

import urllib

url = 'http://test.com/s?wd=哈哈'

url = url.decode('gbk', 'replace')

print urllib.quote(url.encode('utf-8', 'replace'))

结果: http%3a%2f%2ftest.com%2fs%3fwd%3d%e5%93%88%e5%93%88

(2) url解码:

import urllib

encoded_url = 'http%3a%2f%2ftest.com%2fs%3fwd%3d%e5%93%88%e5%93%88'

print urllib.unquote(encoded_url).decode('utf-8', 'replace').encode('gbk', 'replace')

 

 

http://blog.csdn.net/yelbosh/article/details/7551584

 

很好的资料(必看(自))

http://blog.javachen.com/2014/05/24/using-scrapy-to-cralw-data/

 

求救:  使用git clone时, 出现以下错误如何解决?
aaa$ git clone https://github.com/python-excel/xlwt.git
Cloning into 'xlwt'...
/usr/libexec/git-core/git-remote-https.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory

zpc@Lenovo-PC:/usr/libexec/git-core$ gdb /usr/libexec/git-core/git-remote-https.exe
(gdb) r
Starting program: /usr/libexec/git-core/git-remote-https.exe
[New Thread 3192.0x19d0]
gdb: unknown target exception 0xc0000135 at 0x77898f05
During startup program exited with code 0xc0000135.

 

 

http://blog.csdn.net/hjy_six/article/details/6862648

 

Windows和cygwin路径的转换

cygwin下的cd命令可以直接使用Windows的路径表示。

注：不要忘了加上单引号，因为\是bash元字符，用于转义。不用上单引号cd命令收到的参数值就不是C:\Windows\System32\drivers\etc，运行报错。

$cygpath -aw '/cygdrive/c/Windows/System32/drivers/etc'

C:\Windows\System32\drivers\etc

$cygpath -au 'C:\Windows\System32\drivers\etc'

/cygdrive/c/Windows/System32/drivers/etc

有cygpath命令来完成转换，相关的选项是：

-a, --absolute        output absolute path

-w, --windows         print Windows form of NAMEs (C:\WINNT)

-u, --unix            (default) print Unix form of NAMEs (/cygdrive/c/winnt)

 

查看浏览器的user-agent

javascript:alert(navigator.userAgent)

firefox

# -*- encoding: gb2312 -*-

import  codecs, sys

#  用codecs提供的open方法来指定打开的文件的语言编码，它会在读取的时候自动转换为内部unicode

bfile  =  codecs.open( " dddd.txt " ,  ' r ' ,  " big5 " )

# bfile = open("dddd.txt", 'r')

ss  =  bfile.read()

bfile.close()

 

VIM正则表达式

http://www.cnblogs.com/RigorosLee/archive/2011/05/13/2045806.html

 

使用lxml前注意事项：先确保html经过了utf-8解码，即code = html.decode('utf-8', 'ignore')，否则会出现解析出错情况。因为中文被编码成utf-8之后变成 '/u2541' 之类的形式，lxml一遇到 “/”就会认为其标签结束。

 

scrapy命令相关

不显示无用的log

scrapy crawl --nolog test