python---异步IO(asyncio)协程
简单了解
在py3中内置了asyncio模块。其编程模型就是一个消息循环。
模块查看:
from .base_events import *
from .coroutines import * #协程模块,可以将函数装饰为协程
from .events import * #事件模块,事件循环和任务调度都将使用到他
from .futures import * #异步并发模块,该模块对task封装了许多方法,代表将来执行或没有执行的任务的结果。它和task上没有本质上的区别
from .locks import * #异步保证资源同步
from .protocols import *
from .queues import *
from .streams import *
from .subprocess import *
from .tasks import * #创建任务,是对协程的封装,可以查看协程的状态。可以将任务集合
from .transports import *
调用步骤:
1.当我们给一个函数添加了async关键字,或者使用asyncio.coroutine装饰器装饰,就会把它变成一个异步函数。
2.每个线程有一个事件循环,主线程调用asyncio.get_event_loop时会创建事件循环, 3.将任务封装为集合asyncio.gather(*args),之后一起传入事件循环中 4.要把异步的任务丢给这个循环的run_until_complete方法,事件循环会安排协同程序的执行。和方法名字一样,该方法会等待异步的任务完全执行才会结束。
简单使用:
import asyncio,time @asyncio.coroutine #设为异步函数
def func1(num):
print(num,'before---func1----')
yield from asyncio.sleep(5)
print(num,'after---func1----') task = [func1(1),func1(2)] if __name__ == "__main__":
begin = time.time()
loop = asyncio.get_event_loop() #进入事件循环
loop.run_until_complete(asyncio.gather(*task)) #将协同程序注册到事件循环中
loop.close()
end = time.time()
print(end-begin)
before---func1----
before---func1----
after---func1----
after---func1----
5.00528621673584
输出结果
定义一个协程(不同于上面的实例)
import asyncio,time async def func1(num): #使用async关键字定义一个协程,协程也是一种对象,不能直接运行,需要加入事件循环中,才能被调用。
print(num,'before---func1----') if __name__ == "__main__":
begin = time.time() coroutine = func1() loop = asyncio.get_event_loop()
loop.run_until_complete(coroutine)
loop.close()
end = time.time()
print(end-begin)
func1() #由于使用async异步关键字,所以不能直接运行
D:/MyPython/day25/mq/multhread.py:15: RuntimeWarning: coroutine 'func1' was never awaited
func1(2)
print(type(func1),type(coroutine)) #<class 'function'> <class 'coroutine'>
我们可以使用send(None)调用协程(这里不这么使用),这里是将协程放入事件循环中进行处理
coroutine = func1()
try:
coroutine.send(None)
except StopIteration:
pass
创建一个任务(对协程进一步封装,可以查看状态等)
协程对象不能直接运行,在注册事件循环的时候,其实是run_until_complete方法将协程包装成为了一个任务(task)对象.
task对象是Future类的子类,保存了协程运行后的状态,用于未来获取协程的结果
run_until_complete方法查看:
class BaseEventLoop(events.AbstractEventLoop): def run_until_complete(self, future):
"""Run until the Future is done. If the argument is a coroutine, it is wrapped in a Task. WARNING: It would be disastrous to call run_until_complete()
with the same coroutine twice -- it would wrap it in two
different Tasks and that can't be good. Return the Future's result, or raise its exception.
"""
self._check_closed() new_task = not futures.isfuture(future)
future = tasks.ensure_future(future, loop=self)
if new_task:
# An exception is raised if the future didn't complete, so there
# is no need to log the "destroy pending task" message
future._log_destroy_pending = False future.add_done_callback(_run_until_complete_cb)
try:
self.run_forever()
except:
if new_task and future.done() and not future.cancelled():
# The coroutine raised a BaseException. Consume the exception
# to not log a warning, the caller doesn't have access to the
# local task.
future.exception()
raise
finally:
future.remove_done_callback(_run_until_complete_cb)
if not future.done():
raise RuntimeError('Event loop stopped before Future completed.') return future.result()
由源码可以知道,在协程注册后会被自动封装为task任务。所以我们不是必须传入task。但是去创建一个task对象,有利于我们理解协程的状态。
import asyncio,time async def func1(num):
print(num,'before---func1----') if __name__ == "__main__":
begin = time.time() coroutine = func1() loop = asyncio.get_event_loop() task = loop.create_task(coroutine) #创建了任务
print(task) #pending loop.run_until_complete(task)
loop.close()
print(task) #finished
end = time.time()
print(end-begin)
对于协程的4种状态:python---协程理解
print(task) #pending
print(getcoroutinestate(coroutine)) loop.run_until_complete(task)
loop.close()
print(task) #finished
print(getcoroutinestate(coroutine))
CORO_CREATED
before---func1----
<Task finished coro=<func1() done, defined at D:/MyPython/day25/mq/multhread.py:> result=None>
CORO_CLOSED
深入了解:关于Task,create_task(),ensure_future都可以用来创建任务,那么应该使用哪个?
条件使用ensure_future,他是最外层函数,其中调用了create_task()方法,功能全面,而Task官方不推荐直接使用
isinstance(task, asyncio.Future)
将会输出True。绑定回调add_done_callback
async def func1(num):
print(num,'before---func1----')
return "recv num %s"%num def callback(future):
print(future.result()) if __name__ == "__main__":
begin = time.time() coroutine1 = func1(1)
loop = asyncio.get_event_loop()
task1=asyncio.ensure_future(coroutine1)
task1.add_done_callback(callback)
loop.run_until_complete(task1)
loop.close()
end = time.time()
print(end-begin)
1 before---func1----
recv num 1
0.004000186920166016
可以看到,coroutine执行结束时候会调用回调函数。并通过参数future获取协程执行的结果。我们创建的task和回调里的future对象,实际上是同一个对象。
我也可以不使用回调函数,单纯获取返回值
当task状态为finished时候,我们可以直接使用result方法(在future模块)获取返回值
async def func1(num):
print(num,'before---func1----')
return "recv num %s"%num if __name__ == "__main__":
begin = time.time() coroutine1 = func1()
loop = asyncio.get_event_loop()
task1=asyncio.ensure_future(coroutine1)
loop.run_until_complete(task1)
print(task1)
print(task1.result())
loop.close()
end = time.time()
print(end-begin)
before---func1----
<Task finished coro=<func1() done, defined at D:/MyPython/day25/mq/multhread.py:> result='recv num 1'>
recv num
0.0030002593994140625
阻塞和await
使用async关键字定义的协程对象,使用await可以针对耗时的操作进行挂起(是生成器中的yield的替代,但是本地协程函数不允许使用),让出当前控制权。协程遇到await,事件循环将会挂起该协程,执行别的协程,直到其他协程也挂起,或者执行完毕,在进行下一个协程的执行
使用asyncio.sleep模拟阻塞操作。
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num if __name__ == "__main__":
begin = time.time() coroutine1 = func1()
coroutine2 = func1()
loop = asyncio.get_event_loop()
task1=asyncio.ensure_future(coroutine1)
task2=asyncio.ensure_future(coroutine2)
tasks = asyncio.gather(*[task1,task2]) #gather可以实现同时注册多个任务,实现并发操作。wait方法使用一致
loop.run_until_complete(tasks)
loop.close()
end = time.time()
print(end-begin)
并发:使用gather或者wait可以同时注册多个任务,实现并发
gather:Return a future aggregating results from the given coroutines or futures. 返回结果
task1=asyncio.ensure_future(coroutine1)
task2=asyncio.ensure_future(coroutine2)
tasks = asyncio.gather(*[task1,task2])
loop.run_until_complete(tasks)
wait:Returns two sets of Future: (done, pending). #返回dones是已经完成的任务,pending是未完成的任务,都是集合类型
task1=asyncio.ensure_future(coroutine1)
task2=asyncio.ensure_future(coroutine2)
tasks = asyncio.wait([task1,task2])
loop.run_until_complete(tasks)
Usage: done, pending = yield from asyncio.wait(fs)
wait是接收一个列表,而后gather是接收一堆任务数据。
两者的返回值也是不同的
协程嵌套,将多个协程封装到一个主协程中
import asyncio,aiohttp async def fetch_async(url):
print(url)
async with aiohttp.ClientSession() as session:
async with session.get(url) as resp:
print(resp.status)
print(await resp.text()) tasks = [fetch_async('http://www.baidu.com/'), fetch_async('http://www.cnblogs.com/ssyfj/')] event_loop = asyncio.get_event_loop()
results = event_loop.run_until_complete(asyncio.gather(*tasks))
event_loop.close()
关于aiohttp模块的协程嵌套,嵌套更加明显
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num async def main():
coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] dones, pendings = await asyncio.wait(tasks) for task in dones: #对已完成的任务集合进行操作
print("Task ret: ",task.result()) if __name__ == "__main__":
begin = time.time() loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
end = time.time()
print(end-begin)
before---func1----
before---func1----
before---func1----
Task ret: recv num
Task ret: recv num
Task ret: recv num
5.000285863876343
也可以直接使用gather直接获取值
results = await asyncio.gather(*tasks)
for result in results:
print("Task ret: ",result)
我们也可以不在main中处理结果,而是返回到主调用方进行处理
async def main():
coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] return await asyncio.gather(*tasks) if __name__ == "__main__":
begin = time.time() loop = asyncio.get_event_loop()
results = loop.run_until_complete(main())
for result in results:
print("Task ret: ",result)
loop.close()
end = time.time()
print(end-begin)
或者使用wait挂起
return await asyncio.wait(tasks) ----------------------------------------------------
dones,pendings = loop.run_until_complete(main())
for task in dones:
print("Task ret: ",task.result())
或者使用asyncio中的as_completed方法
Return an iterator whose values are coroutines. #返回一个可迭代的协程函数值 When waiting for the yielded coroutines you'll get the results (or
exceptions!) of the original Futures (or coroutines), in the order
in which and as soon as they complete.
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num async def main():
coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] for task in asyncio.as_completed(tasks):
result = await task
print("Task ret: ",result) if __name__ == "__main__":
begin = time.time() loop = asyncio.get_event_loop()
loop.run_until_complete(main()) loop.close()
end = time.time()
print(end-begin)
协程停止
future对象有几个状态:
- Pending
- Running
- Done
- Cacelled
创建future的时候,task为pending,
事件循环调用执行的时候当然就是running,
调用完毕自然就是done,
如果需要停止事件循环,就需要先把task取消。
可以使用asyncio.Task获取事件循环的task
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num if __name__ == "__main__":
begin = time.time() coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] loop = asyncio.get_event_loop()
try:
loop.run_until_complete(asyncio.wait(tasks))
except KeyboardInterrupt as e:
print(asyncio.Task.all_tasks())
for task in asyncio.Task.all_tasks(): #获取所有任务
print(task.cancel()) #单个任务取消
loop.stop() #需要先stop循环
loop.run_forever() #需要在开启事件循环
finally:
loop.close() #统一关闭
end = time.time()
print(end-begin)
before---func1----
before---func1----
before---func1----
{<Task pending coro=<func1() running at multhread.py:> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<loc
als>._on_completion() at C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\asyncio\tasks.py:]>, <
Task pending coro=<wait() running at C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\asyncio\tasks
.py:> wait_for=<Future pending cb=[Task._wakeup()]>>, <Task pending coro=<func1() running at multhread.py:> wait
_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at C:\Users\Administrator\AppData\Loca
l\Programs\Python\Python35\lib\asyncio\tasks.py:]>, <Task pending coro=<func1() running at multhread.py:> wait_f
or=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at C:\Users\Administrator\AppData\Local\
Programs\Python\Python35\lib\asyncio\tasks.py:]>} #未处理,刚刚挂起为pending状态
True #返回True,表示cancel取消成功
True
True
True
3.014172315597534
True表示cannel成功,loop stop之后还需要再次开启事件循环,最后在close,不然还会抛出异常:
Task was destroyed but it is pending!
因为cancel后task的状态依旧是pending
for task in asyncio.Task.all_tasks():
print(task)
print(task.cancel())
print(task)
<Task pending coro=<func1() running at multhread.py:> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<loca
ls>._on_completion() at C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\asyncio\tasks.py:]>
True
<Task pending coro=<func1() running at multhread.py:> wait_for=<Future cancelled> cb=[_wait.<locals>._on_completion
() at C:\Users\Administrator\AppData\Local\Programs\Python\Python35\lib\asyncio\tasks.py:]>
或者使用协程嵌套,main协程相当于最外层task,处理main函数即可
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num async def main():
coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] dones,pendings=await asyncio.wait(tasks) for task in dones:
print("Task ret: ",task.result()) if __name__ == "__main__":
begin = time.time() loop = asyncio.get_event_loop()
task = asyncio.ensure_future(main())
try:
loop.run_until_complete(task)
except KeyboardInterrupt as e:
print(asyncio.gather(*asyncio.Task.all_tasks()).cancel()) #我们只是把上面的单个写成了所有任务集合取消,和协程嵌套关系不大。上面也可以这样写。不过协程嵌套可以简化代码
loop.stop()
loop.run_forever()
finally:
loop.close()
end = time.time()
print(end-begin)
before---func1----
before---func1----
before---func1----
<class 'asyncio.tasks._GatheringFuture'>
True
3.008172035217285
感觉有点多余...
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num if __name__ == "__main__":
begin = time.time() coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] loop = asyncio.get_event_loop()
try:
loop.run_until_complete(asyncio.wait(tasks))
except KeyboardInterrupt as e:
print(asyncio.gather(*tasks).cancel())
loop.stop()
loop.run_forever()
finally:
loop.close()
end = time.time()
print(end-begin)
before---func1----
before---func1----
before---func1----
True
3.008171796798706
上面讨论的都是在同一线程下的事件循环,下面来谈谈不同线程的事件循环
在当前线程中创建一个事件循环(不启用,单纯获取标识),开启一个新的线程,在新的线程中启动事件循环。在当前线程依据事件循环标识,可以向事件中添加协程对象。当前线程不会由于事件循环而阻塞了。
上面在一个线程中执行的事件循环,只有我们主动关闭事件close,事件循环才会结束,会阻塞。
同一线程:
import asyncio,time async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num if __name__ == "__main__":
begin = time.time() coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() tasks = [
asyncio.ensure_future(coroutine1),
asyncio.ensure_future(coroutine2),
asyncio.ensure_future(coroutine3),
] loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
loop.run_forever()
end = time.time()
print(end-begin)
不同线程事件循环(不涉及协程):
import asyncio,time,threading def func1(num):
print(num,'before---func1----')
time.sleep(num)
return "recv num %s"%num def start_loop(loop):
asyncio.set_event_loop(loop)
loop.run_forever() if __name__ == "__main__":
begin = time.time() new_loop = asyncio.new_event_loop() #在当前线程下创建时间循环,(未启用)
t = threading.Thread(target=start_loop,args=(new_loop,)) #开启新的线程去启动事件循环
t.start() new_loop.call_soon_threadsafe(func1,)
new_loop.call_soon_threadsafe(func1,)
new_loop.call_soon_threadsafe(func1,) end = time.time()
print(end-begin) #当前线程未阻塞,耗时0.02800154685974121
before---func1----
0.02800154685974121
before---func1----
before---func1----
新线程协程:
import asyncio,time,threading async def func1(num):
print(num,'before---func1----')
await asyncio.sleep(num)
return "recv num %s"%num def start_loop(loop):
asyncio.set_event_loop(loop)
loop.run_forever() if __name__ == "__main__":
begin = time.time() coroutine1 = func1()
coroutine2 = func1()
coroutine3 = func1() new_loop = asyncio.new_event_loop() #在当前线程下创建时间循环,(未启用)
t = threading.Thread(target=start_loop,args=(new_loop,)) #开启新的线程去启动事件循环
t.start() asyncio.run_coroutine_threadsafe(coroutine1,new_loop) #传参必须是协程对象
asyncio.run_coroutine_threadsafe(coroutine2,new_loop)
asyncio.run_coroutine_threadsafe(coroutine3,new_loop) end = time.time()
print(end-begin) #当前线程未阻塞,耗时0.
before---func1----
before---func1----
before---func1----
0.010000467300415039
主线程通过run_coroutine_threadsafe新注册协程对象。这样就能在子线程中进行事件循环的并发操作,同时主线程又不会被block。
python---异步IO(asyncio)协程的更多相关文章
- Python异步IO之协程(一):从yield from到async的使用
引言:协程(coroutine)是Python中一直较为难理解的知识,但其在多任务协作中体现的效率又极为的突出.众所周知,Python中执行多任务还可以通过多进程或一个进程中的多线程来执行,但两者之中 ...
- Python异步IO之协程(二):使用asyncio的不同方法实现协程
引言:在上一章中我们介绍了从yield from的来源到async的使用,并在最后以asyncio.wait()方法实现协程,下面我们通过不同控制结构来实现协程,让我们一起来看看他们的不同作用吧- 在 ...
- 进程&线程(三):外部子进程subprocess、异步IO、协程、分布式进程
1.外部子进程subprocess python之subprocess模块详解--小白博客 - 夜风2019 - 博客园 python subprocess模块 - lincappu - 博客园 之前 ...
- 异步IO(协程,消息循环队列)
同步是CPU自己主动查看IO操作是否完成,异步是IO操作完成后发出信号通知CPU(CPU是被通知的) 阻塞与非阻塞的区别在于发起IO操作之后,CPU是等待IO操作完成再进行下一步操作,还是不等待去做其 ...
- day41 - 异步IO、协程
目录 (见右侧目录栏导航) - 1. 前言- 2. IO的五种模型- 3. 协程 - 3.1 协程的概念- 4. Gevent 模块 - 4.1 gevent 基本使用 - 4.2 ...
- 异步IO和协程
1-1.并行:真的多任务执行(CPU核数>=任务数):即在某个时刻点上,有多个程序同时运行在多个CPU上 1-2.并发:假的多任务执行(CPU核数<任务数):即一段时间内,有多个程序在同一 ...
- python并发编程之asyncio协程(三)
协程实现了在单线程下的并发,每个协程共享线程的几乎所有的资源,除了协程自己私有的上下文栈:协程的切换属于程序级别的切换,对于操作系统来说是无感知的,因此切换速度更快.开销更小.效率更高,在有多IO操作 ...
- python异步IO编程(一)
python异步IO编程(一) 基础概念 协程:python generator与coroutine 异步IO (async IO):一种由多种语言实现的与语言无关的范例(或模型). asyncio ...
- python异步IO编程(二)
python异步IO编程(二) 目录 开门见山 Async IO设计模式 事件循环 asyncio 中的其他顶层函数 开门见山 下面我们用两个简单的例子来让你对异步IO有所了解 import asyn ...
随机推荐
- Thrift IDL使用方式
I.背景 众所周知,Thrift是一个RPC的框架,其可用于不同语言之间的服务相互调用.比如最近接触到的一个运用环境: *前端使用Node.Js重构了部分我们的老旧代码(前后端未分离的SpringBo ...
- Java常用类之Math类
Java 的常用类Math类: java.lang.Math 提供了系列的静态方法用于科学计算,其方法的参数和返回值类型一般为 double 类型. 如: 1. public static final ...
- (三)java字符串
不可变字符串 Java没有字符串类型,而是提供了一个预定义类String. java中的字符串是不可变字符串,因此无法更改某一个字符串变量的内容. 优点:编译器可以让字符串共享.当复制一个字符串时,原 ...
- javaweb引用jar类库的问题
JAVAWEB中,需要引用的jar类库必须放在/WebContent/WEB-INF/lib文件夹中.不然就会各种报错.
- C# Winform Excel的导出,根据excel模板导出数据
namespace dxhbskymDemo { public partial class ExcelForm : DevExpress.XtraEditors.XtraForm { public E ...
- 3dContactPointAnnotationTool开发日志(四)
没办法,为了能在寝室接着做这玩意只好又在电脑上装一个和实验室版本一样的unity了.虽然打开后UI界面还是一团糟,不过至少要的东西都在,又手动调了调UI界面. 然后把旋转视角功能加上了.鼠标右 ...
- 数论的欧拉定理证明 & 欧拉函数公式(转载)
欧拉函数 :欧拉函数是数论中很重要的一个函数,欧拉函数是指:对于一个正整数 n ,小于 n 且和 n 互质的正整数(包括 1)的个数,记作 φ(n) . 完全余数集合:定义小于 n 且和 n 互质的数 ...
- [OS] 生产者-消费者问题(有限缓冲问题)
·最简单的情形--(一个生产者 + 一个消费者 + 一个大小为1的有限缓冲) 首先来分析其中的同步关系: ·必须在生产者放入一个产品之后,消费者才能够从缓冲中取出产品来消费.·只有在消费者从缓冲区中取 ...
- BZOJ4850/BZOJ2216 JSOI2016灯塔/Poi2011Lightning Conductor(决策单调性)
即对每个i最大化hj-hi+sqrt(|i-j|).先把绝对值去掉,正反各做一次即可.注意到当x>y时,sqrt(x+1)-sqrt(x)<sqrt(y+1)-sqrt(y),所以若对于i ...
- 转:关于Latent Dirichlet Allocation及Hierarchical LDA模型的必读文章和相关代码
关于Latent Dirichlet Allocation及Hierarchical LDA模型的必读文章和相关代码 转: http://andyliuxs.iteye.com/blog/105174 ...