Python3 线程/进程池 concurrent.futures

python3之concurrent.futures一个多线程多进程的直接对接模块，python3.2有线程池了

Python标准库为我们提供了threading和multiprocessing模块编写相应的多线程/多进程代码。从Python3.2开始，标准库为我们提供了concurrent.futures模块，它提供了ThreadPoolExecutor和ProcessPoolExecutor两个类，实现了对threading和multiprocessing的更高级的抽象，对编写线程池/进程池提供了直接的支持。
concurrent.futures基础模块是executor和future。
 
Executor
Executor是一个抽象类，它不能被直接使用。它为具体的异步执行定义了一些基本的方法。
ThreadPoolExecutor和ProcessPoolExecutor继承了Executor，分别被用来创建线程池和进程池的代码。

class Executor(object):
    """This is an abstract base class for concrete asynchronous executors."""
 
    def submit(self, fn, *args, **kwargs):
        raise NotImplementedError()
 
    def map(self, fn, *iterables, timeout=None):
        if timeout is not None:
            end_time = timeout + time.time()
 
        fs = [self.submit(fn, *args) for args in zip(*iterables)]
        def result_iterator():
            try:
                for future in fs:
                    if timeout is None:
                        yield future.result()
                    else:
                        yield future.result(end_time - time.time())
            finally:
                for future in fs:
                    future.cancel()
        return result_iterator()
 
    def shutdown(self, wait=True):
        pass
 
    def __enter__(self):
        return self
 
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.shutdown(wait=True)
        return False
　　
submit()方法
Executor中定义了submit()方法，这个方法的作用是提交一个可执行的回调task，并返回一个future实例。future对象代表的就是给定的调用。
通过下面的例子来理解submit对线程池/进程池的操作。

# coding: utf-8
 
from concurrent.futures import ThreadPoolExecutor
import time
 
 
def return_future(msg):
    time.sleep(3)
    return msg
 
 
# 创建一个线程池
pool = ThreadPoolExecutor(max_workers=2)
 
# 往线程池加入2个task
f1 = pool.submit(return_future, 'hello')
f2 = pool.submit(return_future, 'world')
 
print(f1.done())
time.sleep(3)
print(f2.done())
 
print(f1.result())
print(f2.result())
　　
改写为进程池形式很简单，把ThreadPoolExecutor替换为ProcessPoolExecutor即可。如果需要提交多个task，可以通过循环多次submit()。
map()方法
除了submit，Exectuor还为我们提供了map方法，这个方法返回一个map(func, *iterables)迭代器，迭代器中的回调执行返回的结果有序的。可以通过下面的例子来理解：

# coding: utf-8
 
from concurrent.futures import ThreadPoolExecutor as Pool
import requests
 
URLS = ['http://www.baidu.com', 'http://qq.com', 'http://sina.com']
 
 
def task(url, timeout=10):
    return requests.get(url, timeout=timeout)
 
 
pool = Pool(max_workers=3)
results = pool.map(task, URLS)
 
for ret in results:
    print('%s, %s' % (ret.url, len(ret.content)))
　　执行结果

http://www.baidu.com/, 2381
http://www.qq.com/, 252160
http://www.sina.com.cn/, 607265
　　
Future
Future可以理解为一个在未来完成的操作，这是异步编程的基础。通常情况下，我们执行io操作，访问url时（如下）在等待结果返回之前会产生阻塞，cpu不能做其他事情，而Future的引入帮助我们在等待的这段时间可以完成其他的操作。

import requests   
 
data = requests.get('http://www.baidu.com').content   
print len(data)
Future实例是由Executor.submit()创建的。Future提供了丰富的方法来处理调用。

# coding: utf-8
from concurrent.futures import ThreadPoolExecutor as Pool
from concurrent.futures import as_completed
import requests
 
URLS = ['http://qq.com', 'http://sina.com', 'http://www.baidu.com', ]
 
 
def task(url, timeout=10):
    return requests.get(url, timeout=timeout)
 
 
with Pool(max_workers=3) as executor:
    future_tasks = [executor.submit(task, url) for url in URLS]
 
    for f in future_tasks:
        if f.running():
            print('%s is running' % str(f))
 
    for f in as_completed(future_tasks):
        try:
            ret = f.done()
            if ret:
                f_ret = f.result()
                print('%s, done, result: %s, %s' % (str(f), f_ret.url, len(f_ret.content)))
        except Exception as e:
            f.cancel()
            print(str(e))
　　结果
<Future at 0x7fc2716e1f60 state=running> is running
<Future at 0x7fc27136d4e0 state=running> is running
<Future at 0x7fc27136d710 state=running> is running
<Future at 0x7fc27136d710 state=finished returned Response>, done, result: http://www.baidu.com/, 2381
<Future at 0x7fc2716e1f60 state=finished returned Response>, done, result: http://www.qq.com/, 252343
<Future at 0x7fc27136d4e0 state=finished returned Response>, done, result: http://www.sina.com.cn/, 602366
从运行结果可以看出，as_completed不是按照URLS列表元素的顺序返回的。这也表明，并发访问不通的url时，没有阻塞。
wait
wait方法接会返回一个tuple(元组)，tuple中包含两个set(集合)，一个是completed(已完成的)另外一个是uncompleted(未完成的)。使用wait方法的一个优势就是获得更大的自由度，它接收三个参数FIRST_COMPLETED, FIRST_EXCEPTION和ALL_COMPLETE，默认设置为ALL_COMPLETED。

# coding: utf-8
from concurrent.futures import ThreadPoolExecutor as Pool
from concurrent.futures import wait
import requests
 
URLS = ['http://qq.com', 'http://sina.com', 'http://www.baidu.com', ]
 
 
def task(url, timeout=10):
    return requests.get(url, timeout=timeout)
 
 
with Pool(max_workers=3) as executor:
    future_tasks = [executor.submit(task, url) for url in URLS]
 
    for f in future_tasks:
        if f.running():
            print('%s is running' % str(f))
 
    results = wait(future_tasks)
    done = results[0]
    for x in done:
        print(x)
　　wait有timeout和return_when两个参数可以设置。
timeout控制wait()方法返回前等待的时间。
return_when决定方法什么时间点返回：如果采用默认的ALL_COMPLETED，程序会阻塞直到线程池里面的所有任务都完成；如果采用FIRST_COMPLETED参数，程序并不会等到线程池里面所有的任务都完成。