笔记-twisted

笔记-twisted

1.      简介

Twisted is an event-driven networking engine written in Python and licensed under the open source ​MIT license. Twisted runs on Python 2 and an ever growing subset also works with Python 3.

1.1.    什么是异步

最初始的程序是阻塞型的，就是一句一句执行，如果没有执行完，需要等待/阻塞，无疑，这样的架构在大部分时候会浪费处理能力，是比较低效的。

进程，线程更多的体现为硬件的资源分割和调度；到了单线程内，因为CPU和其它组件/网络的速度差异，仍然会空置大量处理能力。

为了提高线程内的效率，有了协程，但是协程仍不够抽象和高效，对复杂事件来说，代码复杂性很高；

其实协程在本文所讨论的情况下最主要的作用说就是存在大批量类似任务时，优先处理不需要等待其它条件的任务；

更进一步，把事件调度，分发机制抽象一下，独立出来，就是常说的异步+回调，事件驱动了；

1.2.    overview

关键概念理解：

Protocol：定义如何处理消息；

Factory：实际处理的组件，可以理解为是工厂化的Protocol；

reactor：循环体，不停循环，接收事件/信号，分发事件/信号至对应处理类；

Application:更上层的结构体，管理多个service/reactor,最顶层的结构

其它相关概念：

defereeds：异步功能的关键,立即返回一个Deferred对象，它是一个承诺，意思是它所包含的任务一定会有一个结果，在有结果时会调用相应处理函数，如果正常则调用callback，异常则调用errback；当然，callback和errback可以是chain。

endpoints:对server,client更精确的操纵；

transport:对连接的描述及操纵。

上述概念理解后，剩下的细节就可以分步填充到框架里了。

除此之外，还有对数据库的异步支持，enterprice.adbapi

------部件理解------

2.      reactor

参考文档：https://twistedmatrix.com/documents/current/api/twisted.internet.reactor.html

reactor实际是一个抽象，具体使用哪一种reactor依赖于平台，当然，也可以手动显示指定。

reactor类型

IReactorCore    IReactorTime    IReactorProcess

IReactorTCP    IReactorSSL    IReactorUDP

IReactorMulticast    IReactorUNIX    IReactorUNIXDatagram

IReactorFDSet    IReactorThreads    IReactorPluggableResolver

每个reactor都有不同的操作和属性：

以最常用的ireactorcore为例（核心）

常用属性有：

1. run()
2. stop()
3. callWhenRunning(callable, *args, **kw)
4. running

其它比较重要的还有消息注册和维护，不过一般用不到这么深，了解一下就可以了。

3.      protocol

Protocol描述了如何以异步的方式处理网络中的事件，下面是一些典型的操作：

Method	logPrefix	Return a prefix matching the class name, to identify log messages related to this protocol instance.
Method	dataReceived	Called whenever data is received.
Method	connectionLost	Called when the connection is shut down.
Method	makeConnection	Make a connection to a transport and a server.
Method	connectionMade	Called when a connection is made.

4.      factory

定义一些操作和持久化数据，在体系中可以理解为protocol的实例化。

class FingerFactory(protocol.ServerFactory):
    protocol = FingerProtocol

def getUser(self, user):
        return b"No such user"

5.     
deferred

5.1.   
deferred

# deferred 演示案例1
#
from twisted.internet import defer
from twisted.internet import task
from twisted.internet import reactor

# 耗时操作的外壳函数，返回一个deferred对象
# 完成之后的处理函数，使用task模拟了一个耗时操作
def time_wasted_wrapper(job_id):
    def on_done():
        print('time-wasted
job' + str(job_id) + 'done!')
        return job_id

print('begin
time-wasted jon' + str(job_id))
    return task.deferLater(reactor,
3, on_done)

# 回调函数
def on_one_job_done(result):
    print('result
plus 1!')
    return result + 1

def all_jobs_done(result):
    print(str(result))
    print(reactor.__class__)
    print('all jobs
are done!')

reactor.stop()
    time.sleep(1)
    print(reactor.running)

# 模拟添加任务
def install_jobs():
    jobs_list = list()
    for i in range(10):
        job = time_wasted_wrapper(i)
        job.addCallback(on_one_job_done)
        jobs_list.append(job)
    deferred_list =
defer.DeferredList(jobs_list)
   
deferred_list.addCallback(all_jobs_done)

def run_module():
    install_jobs()
    print('all jobs
have started!')
    reactor.run()
    print('www')
    print(reactor.running)

if __name__ == '__main__':
    run_module()
    # reactor.run()

5.2.   
defer.inlineCallbacks

下面的代码演示了defer.inlineCallbacks的用法

# defer

# @defer.inlineCallbacks
#
from twisted.internet.defer import inlineCallbacks,
Deferred, returnValue
from twisted.python.failure import Failure
from twisted.internet import reactor, defer

def loadRemoteData(callback):
    import time
    time.sleep(1)
    callback(1)

def loadRemoteData2(callback):
    import time
    time.sleep(1)
    callback(2)

@defer.inlineCallbacks
def getRemoteData():
    d1 = defer.Deferred()
    print('start
execute r1!')
    reactor.callInThread(loadRemoteData,
d1.callback)
    r1 = yield d1
    print('r1 =',r1)
    d2 = defer.Deferred()
    print('start
execute r2!')
    reactor.callInThread(loadRemoteData2,
d2.callback)
    r2 = yield d2
    print('r2 =', r2)

returnValue(r1 + r2)

def getResult(v):
    print ("result=", v)

if __name__ == '__main__':
    d = getRemoteData()
    d.addCallback(getResult)

reactor.callLater(4,
reactor.stop);
    reactor.run()

6.     
twisted理解-代码版

下面是一个逐步添加功能的twisted代码示例，能方便的理解twisted各个组件的作用及关系。

'''
# 1 基础的事务循环
# 创建了一个循环，没有监听任何端口
from twisted.internet import reactor
reactor.run()
'''

'''
# 2 进一步，声明endpoint，绑定，实现监听1079端口
from twisted.internet import protocol, reactor, endpoints
class FingerProtocol(protocol.Protocol):
    pass

class FingerFactory(protocol.ServerFactory):
    protocol = FingerProtocol

fingerEndpoint = endpoints.serverFromString(reactor, "tcp:1079")
fingerEndpoint.listen(FingerFactory())
reactor.run()
'''

'''
# 3 protocol丰富一些了，会读取一个输入<user>，并返回信息，然后中断连接
# LineReceiver是一个基本的protocol，它有一些方法，具体可查看api

from twisted.internet import protocol, reactor, endpoints
from twisted.protocols import basic

class FingerProtocol(basic.LineReceiver):
    def lineReceived(self, user):
        self.transport.write(self.factory.getUser(user)+
b"\r\n")
        self.transport.loseConnection()

class FingerFactory(protocol.ServerFactory):
    protocol = FingerProtocol

def getUser(self, user):
        return b"No such user"

fingerEndpoint = endpoints.serverFromString(reactor, "tcp:1079")
fingerEndpoint.listen(FingerFactory())
reactor.run()
'''

'''
# 4 factory丰富一些了，有了一些方法和属性
# Read username, output from non-empty factory, drop connections

from twisted.internet import protocol, reactor, endpoints
from twisted.protocols import basic

class FingerProtocol(basic.LineReceiver):
    def lineReceived(self, user):
       
self.transport.write(self.factory.getUser(user) + b"\r\n")
        self.transport.loseConnection()

class FingerFactory(protocol.ServerFactory):
    protocol = FingerProtocol

def __init__(self, users):
        self.users = users

def getUser(self, user):
        return self.users.get(user,
b"No such user")

fingerEndpoint = endpoints.serverFromString(reactor, "tcp:1079")
fingerEndpoint.listen(FingerFactory({ b'moshez' : b'Happy and well'}))
reactor.run()
'''

'''
# 5 加入了deferreds，并为它声明了callback和errback
# Read username, output from non-empty factory, drop connections
# Use deferreds, to minimize synchronicity assumptions

from twisted.internet import protocol, reactor, defer, endpoints
from twisted.protocols import basic

class FingerProtocol(basic.LineReceiver):
    def lineReceived(self, user):
        d = self.factory.getUser(user)

def onError(err):
            return 'Internal error in
server'
        d.addErrback(onError)

def writeResponse(message):
            self.transport.write(message
+ b'\r\n')
           
self.transport.loseConnection()
        d.addCallback(writeResponse)

class FingerFactory(protocol.ServerFactory):
    protocol = FingerProtocol

def __init__(self, users):
        self.users = users

def getUser(self, user):
        return
defer.succeed(self.users.get(user, b"No such user"))

fingerEndpoint = endpoints.serverFromString(reactor, "tcp:1079")
fingerEndpoint.listen(FingerFactory({b'moshez': b'Happy and well'}))
reactor.run()
'''

'''
# 6 application 使用

from twisted.application import service, strports
from twisted.internet import protocol, reactor, defer
from twisted.protocols import basic

class FingerProtocol(basic.LineReceiver):
    def lineReceived(self, user):
        d = self.factory.getUser(user)

def onError(err):
            return 'Internal error in
server'
        d.addErrback(onError)

def writeResponse(message):
            self.transport.write(message
+ b'\r\n')
           
self.transport.loseConnection()
        d.addCallback(writeResponse)

class FingerFactory(protocol.ServerFactory):
    protocol = FingerProtocol

def __init__(self, users):
        self.users = users

def getUser(self, user):
        return
defer.succeed(self.users.get(user, b"No such user"))

application = service.Application('finger', uid=1, gid=1)
factory = FingerFactory({b'moshez': b'Happy and well'})
strports.service("tcp:79", factory,
reactor=reactor).setServiceParent(
   
service.IServiceCollection(application))

'''

7.     
参考文档：

https://twistedmatrix.com/documents/current/core/howto/tutorial/intro.html