笔记-python -asynio

笔记-python -asynio

1.      简介

asyncio是做什么的？

asyncio is a library to write concurrent code using the async/await syntax.

asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc.

asyncio is often a perfect fit for IO-bound and high-levelstructured network code.

python3.0时代，标准库里的异步网络模块：select(非常底层) ，python3.0时代，第三方异步网络库：Tornado，python3.4时代，asyncio：支持TCP,子进程。

现在的asyncio，有了很多的模块已经在支持：

参考文档：python-3.6.5-docs-html/library/asyncio.html?highlight=async#module-asyncio

注：实验环境为python3.6.4，在3.7中asyncio有一定的新特性。

其实python3.6.5文档中asyncio资料学习曲线更人性化，看完它再看新版理解起来也很快。

2.      base eventloop

code:Lib/asyncio/events.py

事件循环的基类，它是核心功能提供者，功能包括：

1. 注册、执行和取消延迟调用；
2. 创建不同客户和服务端联系；
3. 与外部组件联系；
4. 将函数放进线程执行；

2.1.    run an event loop

一些行为和状态获取方法：

1. AbstractEventLoop.run_forever()

Run until stop() is called.

1. AbstractEventLoop.run_until_complete(future)

Run until the Future is done.

1. AbstractEventLoop.is_running()

Returns running status of event loop.

1. AbstractEventLoop.stop()

Stop running the event loop.

1. AbstractEventLoop.is_closed()

Returns True if the event loop was closed.

1. AbstractEventLoop.close()

Close the event loop. The loop must not be running. Pending callbacks will be lost.

This clears the queues and shuts down the executor, but does not wait for the executor to finish.

This is idempotent and irreversible. No other methods should be called after this one.

1. coroutine AbstractEventLoop.shutdown_asyncgens()

Schedule all currently open asynchronous generator objects to close with an aclose() call. After calling this method, the event loop will issue a warning whenever a new asynchronous generator is iterated. Should be used to finalize all scheduled asynchronous generators reliably. Example:

3.      event loops

code: Lib/asyncio/events.py

主要描述循环部分，核心是循环策略。

3.1.    event loop functions

下面的函数是对全局策略的快捷访问，提供了默认选项。

asyncio.get_event_loop()

Equivalent to calling get_event_loop_policy().get_event_loop().

asyncio.set_event_loop(loop)

Equivalent to calling get_event_loop_policy().set_event_loop(loop).

asyncio.new_event_loop()

Equivalent to calling get_event_loop_policy().new_event_loop().

3.2.    事件循环策略

class asyncio.SelectorEventLoop

Event loop based on the selectors module. Subclass of AbstractEventLoop.

Use the most efficient selector available on the platform.

On Windows, only sockets are supported (ex: pipes are not supported): see the MSDN documentation of select.

class asyncio.ProactorEventLoop

Proactor event loop for Windows using “I/O Completion Ports” aka IOCP. Subclass of AbstractEventLoop.

Availability: Windows.

一般用户是不需要进行policy设置的。

class asyncio.AbstractEventLoopPolicy是策略类；

下面的函数可以查改全局策略：

asyncio.get_event_loop_policy()

Get the current event loop policy.

asyncio.set_event_loop_policy(policy)

Set the current event loop policy. If policy is None, the default policy is restored.

# 测试代码：

loop_policy = asyncio.get_event_loop_policy()
print(loop_policy)

# 输出：

<asyncio.windows_events._WindowsDefaultEventLoopPolicy
object at 0x0000003135046898>

4.     
tasks and coroutines

Source code: Lib/asyncio/tasks.py

Source code: Lib/asyncio/coroutines.py

4.1.   
coroutines

有两种声明方式：

1. async def # 在3.5中新增
2. generators 使用生成器应该用@asyncio.coroutine装饰

The word “coroutine”, like the word
“generator”, is used for two different (though related) concepts:

1. The function that defines a
   coroutine (a function definition using async def or decorated
   with @asyncio.coroutine). If disambiguation is needed we will call this
   a coroutine function (iscoroutinefunction() returns True).
2. The object obtained by calling
   a coroutine function. This object represents a computation or an I/O operation
   (usually a combination) that will complete eventually. If disambiguation is
   needed we will call it a coroutine object (iscoroutine() returns True).

要做的事封装进一个函数(coroutine function)。

@asyncio.coroutine

它是一个修饰器，可能允许生成器使用yield from 调用asnyc def 协程，也可以允许生成器被async def 协程调用。但两者没必要同时使用。

代码示例：

@asyncio.coroutine
def hello5(name):
    print(name)
    yield  from asyncio.sleep(2)
    return

print(hello5)
print(hello5('ttt'))
loop = asyncio.get_event_loop()
loop.run_until_complete(hello5('uuuu'))
print(asyncio.sleep(2))

如果没有通过asyncio的接口调用协程，会导致异常

协程对象中可以做的事:

1. result = await future or result = yield from future –
   suspends the coroutine until the future is done, then returns the future’s
   result, or raises an exception, which will be propagated. (If the future is
   cancelled, it will raise a CancelledErrorexception.) Note that tasks are
   futures, and everything said about futures also applies to tasks.
2. result = await coroutine or result = yield from coroutine –
   wait for another coroutine to produce a result (or raise an exception, which
   will be propagated). The coroutine expression must be
   a call to another coroutine.
3. return expression –
   produce a result to the coroutine that is waiting for this one using await or yield from.
4. raise exception –
   raise an exception in the coroutine that is waiting for this one using await or yield from.

Calling a coroutine does not start its code
running – the coroutine object returned by the call doesn’t do anything until
you schedule its execution. There are two basic ways to start it running:
call await coroutine or yield from coroutine from
another coroutine (assuming the other coroutine is already running!), or
schedule its execution using the ensure_future() function or the AbstractEventLoop.create_task() method.

Coroutines (and tasks) can only run when
the event loop is running.

下面结合一些案例理解

import asyncio

async def hello_world():

print("Hello
World!")

loop
= asyncio.get_event_loop()

#
Blocking call which returns when the hello_world() coroutine is done

loop.run_until_complete(hello_world())

loop.close()

这个很简单，声明一个协程，获取循环，把协程放进循环运行，关闭循环。

import asyncio

import datetime

async def display_date(loop):

end_time = loop.time() + 5.0

while True:

print(datetime.datetime.now())

if (loop.time() + 1.0) >= end_time:

break

await asyncio.sleep(1)

loop
= asyncio.get_event_loop()

#
Blocking call which returns when the display_date() coroutine is done

loop.run_until_complete(display_date(loop))

loop.close()

每秒打一个日期值，持续5秒。

案例：

# 协程之间的切换

import asyncio

async def compute(x, y):

print("Compute %s + %s ..." % (x, y))

await asyncio.sleep(1.0)

return x + y

async def print_sum(x, y):

result = await compute(x,
y)

print("%s + %s = %s" % (x, y, result))

loop
= asyncio.get_event_loop()

loop.run_until_complete(print_sum(1, 2))

loop.close()

执行顺序图如下：

4.2.   
future

class asyncio.Future(*, loop=None)

This
class is almost compatible with concurrent.futures.Future.

Differences:

result() and exception() do
not take a timeout argument and raise an exception when the future isn’t done
yet.

Callbacks
registered with add_done_callback() are always called via the event
loop’s call_soon().

This
class is not compatible with
the wait() and as_completed() functions in
the concurrent.futures package.

This
class is not thread safe.

方法：

1. cancel()

Cancel the
future and schedule callbacks.

If the future
is already done or cancelled, return False. Otherwise, change the future’s
state to cancelled, schedule the callbacks and return True.

1. cancelled()

Return True if
the future was cancelled.

1. done()

Return True if
the future is done.

Done means
either that a result / exception are available, or that the future was
cancelled.

1. result()

Return the
result this future represents.

If the future
has been cancelled, raises CancelledError. If the future’s result isn’t
yet available, raises InvalidStateError. If the future is done and has an
exception set, this exception is raised.

1. exception()

Return the
exception that was set on this future.

The exception
(or None if no exception was set) is returned only if the future is
done. If the future has been cancelled, raises CancelledError. If the
future isn’t done yet, raises InvalidStateError.

1. add_done_callback(fn)

Add a callback
to be run when the future becomes done.

The callback
is called with a single argument - the future object. If the future is already
done when this is called, the callback is scheduled with call_soon().

Use
functools.partial to pass parameters to the callback. For example,fut.add_done_callback(functools.partial(print, "Future:", flush=True)) will
call print("Future:", fut,flush=True).

1. remove_done_callback(fn)

Remove all
instances of a callback from the “call when done” list.

Returns the
number of callbacks removed.

1. set_result(result)

Mark the
future done and set its result.

If the future
is already done when this method is called, raises InvalidStateError.

1. set_exception(exception)

Mark the
future done and set an exception.

If the future is already
done when this method is called, raises InvalidStateError.

案例：

import asyncio

async def slow_operation(future):

await asyncio.sleep(1)

future.set_result('Future is done!')

loop
= asyncio.get_event_loop()

future
= asyncio.Future()

asyncio.ensure_future(slow_operation(future))

loop.run_until_complete(future)

print(future.result())

loop.close()

案例：

使用了run_forever及stop。

import asyncio

async def slow_operation(future):

await asyncio.sleep(1)

future.set_result('Future is done!')

def got_result(future):

print(future.result())

loop.stop()

loop
= asyncio.get_event_loop()

future
= asyncio.Future()

asyncio.ensure_future(slow_operation(future))

future.add_done_callback(got_result)

try:

loop.run_forever()

finally:

loop.close()

4.3.   
task

class asyncio.Task(coro, *, loop=None)

包装协程，是future的子类。

task是一个概念，是协程进一步的分解或调度。

取消task并不意味着取消future，但是取消task可能会导致future异常；

基本上task是一个ascio内部调度用的概念，不要直接创建task实例，使用ensure_future，或者AbstractEventLoop.create_task()

此类线程不是线程安全的。

方法：

1. classmethod all_tasks(loop=None)

Return a set of all tasks for an event loop.

By default all tasks for the current event loop are
returned.

1. classmethod current_task(loop=None)

Return the currently running task in an event loop
or None.

By default the current task for the current event loop
is returned.

None is returned when called not in the context of
a Task.

1. cancel()

Request that this task cancel itself.

This arranges for a CancelledError to be
thrown into the wrapped coroutine on the next cycle through the event loop. The
coroutine then has a chance to clean up or even deny the request using
try/except/finally.

Unlike Future.cancel(), this does not guarantee
that the task will be cancelled: the exception might be caught and acted upon,
delaying cancellation of the task or preventing cancellation completely. The
task may also return a value or raise a different exception.

Immediately after this method is
called, cancelled() will not return True (unless the task
was already cancelled). A task will be marked as cancelled when the wrapped
coroutine terminates with a CancelledError exception (even
if cancel() was not called).

1. get_stack(*, limit=None)

Return the list of stack frames for this task’s
coroutine.

1. print_stack(*, limit=None, file=None)
2. Print the stack or traceback
   for this task’s coroutine.

4.4.   
task functions

asyncio.as_completed(fs, *, loop=None, timeout=None)

Return an iterator whose values, when
waited for, are Future instances.

Raises asyncio.TimeoutError if
the timeout occurs before all Futures are done.

Example:

for f in as_completed(fs):

result = yield from f  #
The 'yield from' may raise

# Use result

asyncio.ensure_future(coro_or_future, *, loop=None)

Schedule the execution of a coroutine
object: wrap it in a future. Return a Task object.

If the argument is a Future, it is returned
directly.

asyncio.gather(*coros_or_futures, loop=None, return_exceptions=False)

Return a future aggregating results from
the given coroutine objects or futures.

asyncio.iscoroutine(obj)

Return True if obj is
a coroutine object, which may be based on a generator or
an async def coroutine.

asyncio.iscoroutinefunction(func)

Return True if func is
determined to be a coroutine function, which may be a decorated generator
function or an asyncdef function.

asyncio.run_coroutine_threadsafe(coro, loop)

Submit a coroutine object to a
given event loop.

Return
a concurrent.futures.Future to access the result.

This function is meant to be called from a
different thread than the one where the event loop is running. Usage:

coroutine asyncio.sleep(delay, result=None, *, loop=None)

Create a coroutine that
completes after a given time (in seconds). If result is provided, it
is produced to the caller when the coroutine completes.

The resolution of the sleep depends on
the granularity
of the event loop.

This function is a coroutine.

coroutine asyncio.wait(futures, *, loop=None, timeout=None, return_when=ALL_COMPLETED)

可以指定return_when参数决定什么时候返回

Wait for the Futures and coroutine objects
given by the sequence futures to complete. Coroutines will be wrapped
in Tasks. Returns two sets of Future: (done, pending).

The sequence futures must not be
empty.

timeout can be used to control the
maximum number of seconds to wait before returning. timeout can be an
int or float. If timeout is not specified or None, there is no
limit to the wait time.

return_when indicates when this
function should return. It must be one of the following constants of the concurrent.futures module:

Constant	Description
FIRST_COMPLETED	The function will return when any future finishes or is cancelled.
FIRST_EXCEPTION	The function will return when any future finishes by raising an exception. If no future raises an exception then it is equivalent toALL_COMPLETED.
ALL_COMPLETED	The function will return when all futures finish or are cancelled.

This function is a coroutine.

Usage:

done, pending = yield from asyncio.wait(fs)

Note

This does not raise asyncio.TimeoutError! Futures that aren’t done
when the timeout occurs are returned in the second set.

coroutine asyncio.wait_for(fut, timeout, *, loop=None)

Wait for the single Future or coroutine
object to complete with timeout. If timeout is None,
block until the future completes.

Coroutine will be wrapped in Task.

Returns result of the Future or coroutine.
When a timeout occurs, it cancels the task and raises asyncio.TimeoutError. To avoid the task
cancellation, wrap it in shield().

If the wait is cancelled, the
future fut is also cancelled.

This function is a coroutine,
usage:

result = yield from asyncio.wait_for(fut,
60.0)

Changed in version 3.4.3: If the wait
is cancelled, the future fut is now also cancelled.

5.     
总结

asyncio的核心概念就是循环，future,task；其它的概念都是依附于这三个概念的。

基本上使用asyncio的过程描述如下：

1. 获取循环，包括指定循环策略；
2. 生成及注册future，添加callback
3. 开始循环
4. 有时需要把主线程留下来做交互，而在另一个线程进行事件循环asyncio.run_coroutine_threadsafe(coro, loop)