MiniCrowler

MiniCrawler

Github Path :

https://github.com/LixinZhang/miniCrowler

Introduction:

• MiniCrawler is a simple web crawler implemented by Python.

• Threadpool tech is used to speed up fetching pages.

• One can config the crawler through modify the file config.py. And start the crawling job using python run.py.

• The webs pages fetched will be stored in pages folder.
• check_status.py helps you check the job's status as following:

Rank            Hostname        Times
----------------------------------------
   1             buaa.edu.cn        40
   2             baixing.com        32
   3             cnblogs.com        29
   4              hao123.com         5
   5           xinhuanet.com         2
   6          visionplaza.cn         2
   7           people.com.cn         2
   8                  org.cn         2
   9                 news.cn         2
  10             most.gov.cn         2

More Detail

You can find more detail in my Chinese Blog. Python 多线程抓取网页