PYTHON文件多线程下载
其实,在一般的文件编程中,这有两个概念要说明:
第一是,下载一个大文件,将这个大文件多为多线程。
第二是,下载N多小文件,将每个线程指定下载多个小文件。
现在实现的是多线程下载一个大文件。
今天完成了一个很有意思的下载图片的功能。想加入多线程功能。
#!/usr/bin/python # -*- coding: utf-8 -*- # filename: paxel.py '''It is a multi-thread downloading tool It was developed follow axel. Author: volans E-mail: volansw [at] gmail.com ''' import sys import os import time import urllib from threading import Thread local_proxies = {'http': 'http://131.139.58.200:8080'} class AxelPython(Thread, urllib.FancyURLopener): '''Multi-thread downloading class. run() is a vitural method of Thread. ''' def __init__(self, threadname, url, filename, ranges=0, proxies={}): Thread.__init__(self, name=threadname) urllib.FancyURLopener.__init__(self, proxies) self.name = threadname self.url = url self.filename = filename self.ranges = ranges self.downloaded = 0 def run(self): '''vertual function in Thread''' try: self.downloaded = os.path.getsize( self.filename ) except OSError: #print 'never downloaded' self.downloaded = 0 # rebuild start poind self.startpoint = self.ranges[0] + self.downloaded # This part is completed if self.startpoint >= self.ranges[1]: print 'Part %s has been downloaded over.' % self.filename return self.oneTimeSize = 16384 #16kByte/time print 'task %s will download from %d to %d \n' % (self.name, self.startpoint, self.ranges[1]) self.addheader("Range", "bytes=%d-%d" % (self.startpoint, self.ranges[1])) self.urlhandle = self.open( self.url ) data = self.urlhandle.read( self.oneTimeSize ) while data: filehandle = open( self.filename, 'ab+' ) filehandle.write( data ) filehandle.close() self.downloaded += len( data ) #print "%s" % (self.name) #progress = u'\r...' data = self.urlhandle.read( self.oneTimeSize ) def GetUrlFileSize(url, proxies={}): urlHandler = urllib.urlopen( url, proxies=proxies ) headers = urlHandler.info().headers length = 0 for header in headers: if header.find('Length') != -1: length = header.split(':')[-1].strip() length = int(length) return length def SpliteBlocks(totalsize, blocknumber): blocksize = totalsize/blocknumber ranges = [] for i in range(0, blocknumber-1): ranges.append((i*blocksize, i*blocksize +blocksize - 1)) ranges.append(( blocksize*(blocknumber-1), totalsize -1 )) return ranges def islive(tasks): for task in tasks: if task.isAlive(): return True return False def paxel(url, output, blocks=6, proxies=local_proxies): ''' paxel ''' size = GetUrlFileSize( url, proxies ) ranges = SpliteBlocks( size, blocks ) threadname = [ "thread_%d" % i for i in range(0, blocks) ] filename = [ "tmpfile_%d" % i for i in range(0, blocks) ] tasks = [] for i in range(0,blocks): task = AxelPython( threadname[i], url, filename[i], ranges[i] ) task.setDaemon( True ) task.start() tasks.append( task ) time.sleep( 2 ) while islive(tasks): downloaded = sum( [task.downloaded for task in tasks] ) process = downloaded/float(size)*100 show = u'\rFilesize:%d Downloaded:%d Completed:%.2f%%' % (size, downloaded, process) sys.stdout.write(show) sys.stdout.flush() time.sleep( 0.1 ) print filehandle = open( output, 'wb+' ) for i in filename: f = open( i, 'rb' ) filehandle.write( f.read() ) f.close() try: os.remove(i) pass except: pass filehandle.close() if __name__ == '__main__': url = "http://6.duote.com.cn/geany_setup.zip" output = 'geany_setup.zip' paxel( url, output, blocks=5, proxies={} )
PYTHON文件多线程下载的更多相关文章
- python文件管道 下载图集
# -*- coding: utf-8 -*- import re from time import sleep import scrapy from scrapy.linkextractors im ...
- Python实现多线程下载
#!/usr/bin/python # -*- coding: utf-8 -*- # filename: paxel.py '''It is a multi-thread downloading t ...
- android 学习随笔十三(网络:多线程下载)
多线程断点续传下载1.多线程:快* 原理:抢占服务器资源* 单线程下载:线程从第0个字节开始下,下到最后一个字节,在本地硬盘的临时文件中从第0个字节开始写,写到最后一个字节,下载完成时,临时文件也写完 ...
- [iOS 多线程 & 网络 - 2.4] - 大文件下载 (边下边写/暂停恢复下载/压缩解压zip/多线程下载)
A.需求 边下边写入硬盘 显示下载进度 暂停/恢复 下载 解压文件 多线程下载 B.基本知识 1.小文件下载 如果文件比较小,下载方式会比较多直接用NSData的+ (id)dataWithCon ...
- Android中多线程下载列表的封装实现(含进度反馈)
来源:http://blog.csdn.net/u011638883/article/details/17347015 实现了一下Android中的文件多线程下载模块,支持自定义线程数.断点续传.下载 ...
- Python之FTP多线程下载文件之分块多线程文件合并
Python之FTP多线程下载文件之分块多线程文件合并 欢迎大家阅读Python之FTP多线程下载系列之二:Python之FTP多线程下载文件之分块多线程文件合并,本系列的第一篇:Python之FTP ...
- Python之FTP多线程下载文件之多线程分块下载文件
Python之FTP多线程下载文件之多线程分块下载文件 Python中的ftplib模块用于对FTP的相关操作,常见的如下载,上传等.使用python从FTP下载较大的文件时,往往比较耗时,如何提高从 ...
- python线程使用场景 多线程下载
http://blog.xiayf.cn/2015/09/11/parallelism-in-one-line http://python.jobbole.com/84327/ http://www. ...
- Python实现多线程HTTP下载器
本文将介绍使用Python编写多线程HTTP下载器,并生成.exe可执行文件. 环境:windows/Linux + Python2.7.x 单线程 在介绍多线程之前首先介绍单线程.编写单线程的思路为 ...
随机推荐
- 红领的短板:线下“O”瓶颈_财经频道_一财网
红领的短板:线下"O"瓶颈_财经频道_一财网 红领的短板:线下"O"瓶颈
- python 查看文件夹权限组和用OS模块操作文件夹
@建议操作server服务器文件夹时可以映射网络驱动盘 import win32securityimport ntsecuritycon as con FILENAME = r'D:\tmp\acc_ ...
- Html5新特性 <canvas>画板画直线
以下样例为用canvas标签画多条直线 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" & ...
- 加入gitignore文件没有起作用怎么办
步骤一: 假设有未提交的文件先提交到Git. 步骤二: 在Git根文件夹下运行以下的Git命令: git rm -r --cached . git add . git commit -m " ...
- MSSQL row_number简单使用语法
MSSQL row_number简单使用语法 select * from ( select row_number() over(partition by threadid order by date ...
- wildcard
[rusky@rhel7 test]$ lstest1 test123 test2 test317 test33 test335 test336 test44 testtest[rus ...
- 综合使用LruCache和DiskLruCache 缓存图片
Activity public class MainActivity extends Activity { private GridView mPhotoWall; private P ...
- mysql存储过程中使用临时表和游标
1.临时表 DROP PROCEDURE IF EXISTS `P_GetMonitorPeople`; CREATE PROCEDURE P_GetMonitorPeople (IN fgid IN ...
- js三种消息框总结-警告框、确认框、提示框
js消息框类别:警告框.确认框.提示框 警告框:alert("文本"); 确认框:confirm("文本"); 提示框:prompt("文本" ...
- nyoj 44
//nyoj 44 //和上面一题一样,求子串和,但是代码非常简洁..... 时间复杂度为n #include <iostream> using namespace std; int ma ...