Python 深入剖析SocketServer模块（二）（V2.7.11）

五、Mix-In混合类

昨天介绍了BaseServer和BaseRequestHandler两个基类，它们只用与派生，所以贴了它们派生的子类代码。

今天介绍两个混合类，ForkingMix-In 和 ThreadingMix-In，两者分别实现了核心的进程化和线程化的功能，如前面简介中所提，作为混合类，它们与服务器类一并使用以提供一些异步特性，Mix-in 这个类必须首先实现，因为它重写了定义UDPServer的方法。注意，它们不会被直接实例化。

5.1 ForkingMixIn

该类针对每一个监听到来的新的连接请求都会fork一个新的子进程，在子进程中完成请求处理。这里需要注意，由于windows对fork支持的不完整，所以无法在windows环境下运行该模块的forkingMixIn类，只能在linux或Mac环境中测试运行。

<span style="font-size:24px;">

class ForkingMixIn:

    timeout = 300
    active_children = None
    max_children = 40</span>

首先定义了超时时间为300，最大的子进程数量为40个。

<span style="font-size:24px;">    

<span style="white-space:pre">	</span>def collect_children(self):
        """Internal routine to wait for children that have exited."""
        if self.active_children is None:
            return

        # If we're above the max number of children, wait and reap them until
        # we go back below threshold. Note that we use waitpid(-1) below to be
        # able to collect children in size(<defunct children>) syscalls instead
        # of size(<children>): the downside is that this might reap children
        # which we didn't spawn, which is why we only resort to this when we're
        # above max_children.
        while len(self.active_children) >= self.max_children:
            try:
                pid, _ = os.waitpid(-1, 0)
                self.active_children.discard(pid)
            except OSError as e:
                if e.errno == errno.ECHILD:
                    # we don't have any children, we're done
                    self.active_children.clear()
                elif e.errno != errno.EINTR:
                    break

        # Now reap all defunct children.
        for pid in self.active_children.copy():
            try:
                pid, _ = os.waitpid(pid, os.WNOHANG)
                # if the child hasn't exited yet, pid will be 0 and ignored by
                # discard() below
                self.active_children.discard(pid)
            except OSError as e:
                if e.errno == errno.ECHILD:
                    # someone else reaped it
                    self.active_children.discard(pid)</span><span style="font-size:18px;">
</span>

collect_children()方法用于判断当前的子进程数是否超过阈值，以保证程序的稳定运行。如果当前的fork的子进程数超过阈值40，我们把主进程阻塞住，使os.waitpid(-1)，直到有子进程处理完成请求并且断开连接，等待总体的正在运行的子进程数降到阈值以下；与此同时，该方法也会通过分配出去的pid遍历所有fork的子进程，查看它们是否在正常工作，如果发现僵死进程或者不存在的子进程，主进程则会调用discard()方法将子进程占用的资源回收，以便分配给其他新到来的请求。

<span style="font-size:24px;">    

<span style="white-space:pre">	</span>def process_request(self, request, client_address):
        """Fork a new subprocess to process the request."""
        self.collect_children()
        pid = os.fork()
        if pid:
            # Parent process
            if self.active_children is None:
                self.active_children = set()
            self.active_children.add(pid)
            self.close_request(request) #close handle in parent process
            return
        else:
            # Child process.
            # This must never return, hence os._exit()!
            try:
                self.finish_request(request, client_address)
                self.shutdown_request(request)
                os._exit(0)
            except:
                try:
                    self.handle_error(request, client_address)
                    self.shutdown_request(request)
                finally:
                    os._exit(1)</span>

process_request()方法为主进程监听连接请求，一旦发现了连接请求，首先调用上面的collect_children()方法，查看set()这个资源池中的子进程数是否达到阈值，如果没有，则为新到来的请求fork一个子进程，分配一个pid放入set()资源池中，然后在子进程中处理到来的请求，注意，子进程中不能有return值，只能用os._exit()退出子进程。为了便于理解，在centos上写了个简单的server和client，当有多个连接请求时，我们    ps -ef | grep sock  ，发现server端有多个子进程。如下图：

很明显，主进程fork了三个子进程处理连接请求，而我也恰好开了三个TCP的连接。此时，我们把一个客户端断开，再ps -ef | grep sock  ，如下图：

此时，我们发现了僵死进程（19385）<defunct>，然而当我把断开的客户端重新启动时，就恢复了3个活跃的子进程，并且进程号增加，说明父进程在新的连接请求到来时清理了set()资源池，把僵死的子进程干掉，并为新的请求分配了新的pid，放入set()资源池中。

5.2 ThreadingMixIn

该类会针对每一个新到来的连接请求分配一个新的线程来处理，这里面有用到python的thread和threading模块，可以回忆之前的这篇文章 ：python--多线程 
 （中间备考隔了太长时间，自己都忘得差不多了，尴尬）

<span style="font-size:24px;">

class ThreadingMixIn:
    """Mix-in class to handle each request in a new thread."""

    # Decides how threads will act upon termination of the
    # main process
    daemon_threads = False</span>

<span style="font-size:24px;">    

def process_request_thread(self, request, client_address):
        """Same as in BaseServer but as a thread.

        In addition, exception handling is done here.

        """
        try:
            self.finish_request(request, client_address)
            self.shutdown_request(request)
        except:
            self.handle_error(request, client_address)
            self.shutdown_request(request)</span>

<span style="font-size:24px;">    

def process_request(self, request, client_address):
        """Start a new thread to process the request."""
        t = threading.Thread(target = self.process_request_thread,
                             args = (request, client_address))
        t.daemon = self.daemon_threads
        t.start()</span>

六、应用举例

以上提到的所有的类，我们最好不要直接实例化，简介里面提到过，我们最好使用它们的混合来使我们的server更加简单强大，我们需要做的就是按照我们的需求重写Handle方法，然后调用以下四种方法之一就可以了：

<span style="font-size:24px;">

class ForkingUDPServer(ForkingMixIn, UDPServer): pass
class ForkingTCPServer(ForkingMixIn, TCPServer): pass

class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass
class ThreadingTCPServer(ThreadingMixIn, TCPServer): pass</span>

下面是服务器端调用ThreadingTCPServer类实例化的一个简单例子。供日后参考：

server端：

# -*- coding:utf-8 -*-
__author__ = 'webber'
import SocketServer
'''
多线程并发实现TCP连接
'''

class MyTCPHandler(SocketServer.BaseRequestHandler):

    def handle(self):
        while True:
            try:
                #self.request is the TCP socket connected to the client
                self.data = self.request.recv(1024).strip()
                print "{}  wrote:".format(self.client_address)
                #client_address[0]:客户端IP
                #client_address[1]:客户端端口号
                print self.data
                #just send back the same data,but upper-cased
                self.request.sendall(self.data.upper())
            except Exception:
                print 'A client has left!!!'
                break

if __name__ == "__main__":
    HOST,PORT = "localhost",9999
    # 把类实例化,把自己写的类绑定到ThreadingTCPServer上
    server = SocketServer.ForkingTCPServer((HOST, PORT), MyTCPHandler)

    # Activate the server; this will keep running until you
    # interrupt the program with Ctrl-C
    '''
    Handle one request at a time until shutdown.

    Polls for shutdown every poll_interval seconds. Ignores
    self.timeout. If you need to do periodic tasks, do them in
    another thread.
    '''
    print 'waiting for connection........ '
    server.serve_forever()

客户端：

# -*- coding:utf-8 -*-
__author__ = 'webber'
import socket

HOST = 'localhost'
PORT = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

try:
    while True:
        msg = raw_input(">>>:").strip()
        if len(msg) is None:
            continue
        s.sendall(msg)
        data = s.recv(1024)
        print "Received: ",data
    s.close()
except socket.error,e:
    print "server disconnected!!!", e