其实,在一般的文件编程中,这有两个概念要说明:

第一是,下载一个大文件,将这个大文件多为多线程。

第二是,下载N多小文件,将每个线程指定下载多个小文件。

现在实现的是多线程下载一个大文件。

今天完成了一个很有意思的下载图片的功能。想加入多线程功能。

#!/usr/bin/python
# -*- coding: utf-8 -*-
# filename: paxel.py

'''It is a multi-thread downloading tool

    It was developed follow axel.
        Author: volans
        E-mail: volansw [at] gmail.com
'''

import sys
import os
import time
import urllib
from threading import Thread

local_proxies = {'http': 'http://131.139.58.200:8080'}

class AxelPython(Thread, urllib.FancyURLopener):
    '''Multi-thread downloading class.

        run() is a vitural method of Thread.
    '''
    def __init__(self, threadname, url, filename, ranges=0, proxies={}):
        Thread.__init__(self, name=threadname)
        urllib.FancyURLopener.__init__(self, proxies)
        self.name = threadname
        self.url = url
        self.filename = filename
        self.ranges = ranges
        self.downloaded = 0

    def run(self):
        '''vertual function in Thread'''
        try:
            self.downloaded = os.path.getsize( self.filename )
        except OSError:
            #print 'never downloaded'
            self.downloaded = 0

        # rebuild start poind
        self.startpoint = self.ranges[0] + self.downloaded

        # This part is completed
        if self.startpoint >= self.ranges[1]:
            print 'Part %s has been downloaded over.' % self.filename
            return

        self.oneTimeSize = 16384 #16kByte/time
        print 'task %s will download from %d to %d \n' % (self.name, self.startpoint, self.ranges[1])
        self.addheader("Range", "bytes=%d-%d" % (self.startpoint, self.ranges[1]))

        self.urlhandle = self.open( self.url )

        data = self.urlhandle.read( self.oneTimeSize )
        while data:
            filehandle = open( self.filename, 'ab+' )
            filehandle.write( data )
            filehandle.close()

            self.downloaded += len( data )
            #print "%s" % (self.name)
            #progress = u'\r...'

            data = self.urlhandle.read( self.oneTimeSize )

def GetUrlFileSize(url, proxies={}):
    urlHandler = urllib.urlopen( url, proxies=proxies )
    headers = urlHandler.info().headers
    length = 0
    for header in headers:
        if header.find('Length') != -1:
            length = header.split(':')[-1].strip()
            length = int(length)
    return length

def SpliteBlocks(totalsize, blocknumber):
    blocksize = totalsize/blocknumber
    ranges = []
    for i in range(0, blocknumber-1):
        ranges.append((i*blocksize, i*blocksize +blocksize - 1))
    ranges.append(( blocksize*(blocknumber-1), totalsize -1 ))

    return ranges
def islive(tasks):
    for task in tasks:
        if task.isAlive():
            return True
    return False

def paxel(url, output, blocks=6, proxies=local_proxies):
    ''' paxel
    '''
    size = GetUrlFileSize( url, proxies )
    ranges = SpliteBlocks( size, blocks )

    threadname = [ "thread_%d" % i for i in range(0, blocks) ]
    filename = [ "tmpfile_%d" % i for i in range(0, blocks) ]

    tasks = []
    for i in range(0,blocks):
        task = AxelPython( threadname[i], url, filename[i], ranges[i] )
        task.setDaemon( True )
        task.start()
        tasks.append( task )

    time.sleep( 2 )
    while islive(tasks):
        downloaded = sum( [task.downloaded for task in tasks] )
        process = downloaded/float(size)*100
        show = u'\rFilesize:%d Downloaded:%d Completed:%.2f%%' % (size, downloaded, process)
        sys.stdout.write(show)
        sys.stdout.flush()
        time.sleep( 0.1 )
        print

    filehandle = open( output, 'wb+' )
    for i in filename:
        f = open( i, 'rb' )
        filehandle.write( f.read() )
        f.close()
        try:
            os.remove(i)
            pass
        except:
            pass

    filehandle.close()

if __name__ == '__main__':
    url = "http://6.duote.com.cn/geany_setup.zip"
    output = 'geany_setup.zip'
    paxel( url, output, blocks=5, proxies={} )

PYTHON文件多线程下载的更多相关文章

  1. python文件管道 下载图集

    # -*- coding: utf-8 -*- import re from time import sleep import scrapy from scrapy.linkextractors im ...

  2. Python实现多线程下载

    #!/usr/bin/python # -*- coding: utf-8 -*- # filename: paxel.py '''It is a multi-thread downloading t ...

  3. android 学习随笔十三(网络:多线程下载)

    多线程断点续传下载1.多线程:快* 原理:抢占服务器资源* 单线程下载:线程从第0个字节开始下,下到最后一个字节,在本地硬盘的临时文件中从第0个字节开始写,写到最后一个字节,下载完成时,临时文件也写完 ...

  4. [iOS 多线程 & 网络 - 2.4] - 大文件下载 (边下边写/暂停恢复下载/压缩解压zip/多线程下载)

    A.需求 边下边写入硬盘 显示下载进度 暂停/恢复 下载 解压文件 多线程下载   B.基本知识 1.小文件下载 如果文件比较小,下载方式会比较多直接用NSData的+ (id)dataWithCon ...

  5. Android中多线程下载列表的封装实现(含进度反馈)

    来源:http://blog.csdn.net/u011638883/article/details/17347015 实现了一下Android中的文件多线程下载模块,支持自定义线程数.断点续传.下载 ...

  6. Python之FTP多线程下载文件之分块多线程文件合并

    Python之FTP多线程下载文件之分块多线程文件合并 欢迎大家阅读Python之FTP多线程下载系列之二:Python之FTP多线程下载文件之分块多线程文件合并,本系列的第一篇:Python之FTP ...

  7. Python之FTP多线程下载文件之多线程分块下载文件

    Python之FTP多线程下载文件之多线程分块下载文件 Python中的ftplib模块用于对FTP的相关操作,常见的如下载,上传等.使用python从FTP下载较大的文件时,往往比较耗时,如何提高从 ...

  8. python线程使用场景 多线程下载

    http://blog.xiayf.cn/2015/09/11/parallelism-in-one-line http://python.jobbole.com/84327/ http://www. ...

  9. Python实现多线程HTTP下载器

    本文将介绍使用Python编写多线程HTTP下载器,并生成.exe可执行文件. 环境:windows/Linux + Python2.7.x 单线程 在介绍多线程之前首先介绍单线程.编写单线程的思路为 ...

随机推荐

  1. 红领的短板:线下“O”瓶颈_财经频道_一财网

    红领的短板:线下"O"瓶颈_财经频道_一财网 红领的短板:线下"O"瓶颈

  2. python 查看文件夹权限组和用OS模块操作文件夹

    @建议操作server服务器文件夹时可以映射网络驱动盘 import win32securityimport ntsecuritycon as con FILENAME = r'D:\tmp\acc_ ...

  3. Html5新特性 <canvas>画板画直线

     以下样例为用canvas标签画多条直线 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" & ...

  4. 加入gitignore文件没有起作用怎么办

    步骤一: 假设有未提交的文件先提交到Git. 步骤二: 在Git根文件夹下运行以下的Git命令: git rm -r --cached . git add . git commit -m " ...

  5. MSSQL row_number简单使用语法

    MSSQL row_number简单使用语法 select * from ( select row_number() over(partition by threadid order by date ...

  6. wildcard

    [rusky@rhel7 test]$ lstest1  test123  test2  test317  test33  test335  test336  test44  testtest[rus ...

  7. 综合使用LruCache和DiskLruCache 缓存图片

    Activity public class MainActivity extends Activity {     private GridView mPhotoWall;     private P ...

  8. mysql存储过程中使用临时表和游标

    1.临时表 DROP PROCEDURE IF EXISTS `P_GetMonitorPeople`; CREATE PROCEDURE P_GetMonitorPeople (IN fgid IN ...

  9. js三种消息框总结-警告框、确认框、提示框

    js消息框类别:警告框.确认框.提示框 警告框:alert("文本"); 确认框:confirm("文本"); 提示框:prompt("文本" ...

  10. nyoj 44

    //nyoj 44 //和上面一题一样,求子串和,但是代码非常简洁..... 时间复杂度为n #include <iostream> using namespace std; int ma ...