大聊Python----Select解析

1、首先列一下，sellect、poll、epoll三者的区别 
select 
select最早于1983年出现在4.2BSD中，它通过一个select()系统调用来监视多个文件描述符的数组，当select()返回后，该数组中就绪的文件描述符便会被内核修改标志位，使得进程可以获得这些文件描述符从而进行后续的读写操作。

select目前几乎在所有的平台上支持，其良好跨平台支持也是它的一个优点，事实上从现在看来，这也是它所剩不多的优点之一。

select的一个缺点在于单个进程能够监视的文件描述符的数量存在最大限制，在Linux上一般为1024，不过可以通过修改宏定义甚至重新编译内核的方式提升这一限制。

另外，select()所维护的存储大量文件描述符的数据结构，随着文件描述符数量的增大，其复制的开销也线性增长。同时，由于网络响应时间的延迟使得大量TCP连接处于非活跃状态，但调用select()会对所有socket进行一次线性扫描，所以这也浪费了一定的开销。

poll 
poll在1986年诞生于System V Release 3，它和select在本质上没有多大差别，但是poll没有最大文件描述符数量的限制。

poll和select同样存在一个缺点就是，包含大量文件描述符的数组被整体复制于用户态和内核的地址空间之间，而不论这些文件描述符是否就绪，它的开销随着文件描述符数量的增加而线性增大。

另外，select()和poll()将就绪的文件描述符告诉进程后，如果进程没有对其进行IO操作，那么下次调用select()和poll()的时候将再次报告这些文件描述符，所以它们一般不会丢失就绪的消息，这种方式称为水平触发（Level Triggered）。

epoll 
直到Linux2.6才出现了由内核直接支持的实现方法，那就是epoll，它几乎具备了之前所说的一切优点，被公认为Linux2.6下性能最好的多路I/O就绪通知方法。

epoll可以同时支持水平触发和边缘触发（Edge Triggered，只告诉进程哪些文件描述符刚刚变为就绪状态，它只说一遍，如果我们没有采取行动，那么它将不会再次告知，这种方式称为边缘触发），理论上边缘触发的性能要更高一些，但是代码实现相当复杂。

epoll同样只告知那些就绪的文件描述符，而且当我们调用epoll_wait()获得就绪文件描述符时，返回的不是实际的描述符，而是一个代表就绪描述符数量的值，你只需要去epoll指定的一个数组中依次取得相应数量的文件描述符即可，这里也使用了内存映射（mmap）技术，这样便彻底省掉了这些文件描述符在系统调用时复制的开销。

另一个本质的改进在于epoll采用基于事件的就绪通知方式。在select/poll中，进程只有在调用一定的方法后，内核才对所有监视的文件描述符进行扫描，而epoll事先通过epoll_ctl()来注册一个文件描述符，一旦基于某个文件描述符就绪时，内核会采用类似callback的回调机制，迅速激活这个文件描述符，当进程调用epoll_wait()时便得到通知。

2、Python select

Python的select()方法直接调用操作系统的IO接口，它监控sockets,open files, and pipes(所有带fileno()方法的文件句柄)何时变成readable 和writeable, 或者通信错误，select()使得同时监控多个连接变的简单，并且这比写一个长循环来等待和监控多客户端连接要高效，因为select直接通过操作系统提供的C的网络接口进行操作，而不是通过Python的解释器。

注意：Using Python’s file objects with select() works for Unix, but is not supported under Windows.

接下来通过echo server例子要以了解select 是如何通过单进程实现同时处理多个非阻塞的socket连接的

## socket_client #3

import socket

HOST = 'HW-20180425SPSL'  # The remote host
PORT = 9000  # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
while True:
    msg = bytes(input(">>:"), encoding="utf8")
    s.sendall(msg)
    data = s.recv(1024)

    #
    print('Received', data)
s.close()

## socket_server ##

import select
import socket
import queue

server = socket.socket()
server.bind(('HW-20180425SPSL',9000))
server.listen(1000)

server.setblocking(False) #不阻塞

msg_dic = {}

inputs = [server,]
#inputs = [server,conn] #[conn,]
#inputs = [server,conn,conn2] #[conn2,]
outputs = [] #
#outputs = [r1,] #
while True:
    readable ,writeable,exceptional= select.select(inputs, outputs, inputs )
    print(readable,writeable,exceptional)
    for r in readable:
        if r is server: #代表来了一个新连接
            conn,addr = server.accept()
            print("来了个新连接",addr)
            inputs.append(conn) #是因为这个新建立的连接还没发数据过来，现在就接收的话程序就报错了，
            #所以要想实现这个客户端发数据来时server端能知道，就需要让select再监测这个conn
            msg_dic[conn] = queue.Queue() #初始化一个队列，后面存要返回给这个客户端的数据
        else: #conn2
            data = r.recv(1024)
            print("收到数据",data)
            msg_dic[r].put(data)

            outputs.append(r) #放入返回的连接队列里
            # r.send(data)
            # print("send done....")

    for w in writeable: #要返回给客户端的连接列表
        data_to_client = msg_dic[w].get()
        w.send(data_to_client) #返回给客户端源数据

        outputs.remove(w) #确保下次循环的时候writeable,不返回这个已经处理完的连接了

    for e in exceptional:
        if e in outputs:
            outputs.remove(e)

        inputs.remove(e)

        del msg_dic[e]

程序执行后的结果为：

socket_client.py

>>:aa
Received b'aa'
>>:cc
Received b'cc'
>>:

socket_server.py

[<socket.socket fd=236, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9000)>] [] []
来了个新连接 ('10.3.9.223', 55065)
[<socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9000), raddr=('10.3.9.223', 55065)>] [] []
收到数据 b'aa'
[] [<socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9000), raddr=('10.3.9.223', 55065)>] []
[<socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9000), raddr=('10.3.9.223', 55065)>] [] []
收到数据 b'cc'
[] [<socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9000), raddr=('10.3.9.223', 55065)>] []

 3、selectors模块

该模块封装了select、epoll，在windows里默认使用select，而在Linux里，默认使用epoll

## socket_client ##

import socket
import sys

messages = [ b'This is the message. ',
             b'It will be sent ',
             b'in parts.',
             ]
server_address = ('HW-20180425SPSL', 9998)

# Create a TCP/IP socket
socks = [ socket.socket(socket.AF_INET, socket.SOCK_STREAM) for i in range(2)]
print(socks)
# Connect the socket to the port where the server is listening
print('connecting to %s port %s' % server_address)
for s in socks:
    s.connect(server_address)

for message in messages:

    # Send messages on both sockets
    for s in socks:
        print('%s: sending "%s"' % (s.getsockname(), message) )
        s.send(message)

    # Read responses on both sockets
    for s in socks:
        data = s.recv(1024)
        print( '%s: received "%s"' % (s.getsockname(), data) )
        if not data:
            print( 'closing socket', s.getsockname() )

## socket_server ##

import selectors
import socket

sel = selectors.DefaultSelector()

def accept(sock, mask):
    conn, addr = sock.accept()  # Should be ready
    print('accepted', conn, 'from', addr,mask)
    conn.setblocking(False)
    sel.register(conn, selectors.EVENT_READ, read) #新连接注册read回调函数

def read(conn, mask):
    data = conn.recv(1024)  # Should be ready
    if data:
        print('echoing', repr(data), 'to', conn)
        conn.send(data)  # Hope it won't block
    else:
        print('closing', conn)
        sel.unregister(conn)
        conn.close()

sock = socket.socket()
sock.bind(('HW-20180425SPSL', 9998))
sock.listen(100)
sock.setblocking(False)
sel.register(sock, selectors.EVENT_READ, accept)

while True:
    events = sel.select() #默认阻塞，有活动连接就返回活动的连接列表
    for key, mask in events:
        callback = key.data #accept
        callback(key.fileobj, mask) #key.fileobj=  文件句柄

程序执行后的结果为：

socket_client.py

[<socket.socket fd=236, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>, <socket.socket fd=248, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>]
connecting to HW-20180425SPSL port 9998
('10.3.9.223', 58356): sending "b'This is the message. '"
('10.3.9.223', 58357): sending "b'This is the message. '"
('10.3.9.223', 58356): received "b'This is the message. '"
('10.3.9.223', 58357): received "b'This is the message. '"
('10.3.9.223', 58356): sending "b'It will be sent '"
('10.3.9.223', 58357): sending "b'It will be sent '"
('10.3.9.223', 58356): received "b'It will be sent '"
('10.3.9.223', 58357): received "b'It will be sent '"
('10.3.9.223', 58356): sending "b'in parts.'"
('10.3.9.223', 58357): sending "b'in parts.'"
('10.3.9.223', 58356): received "b'in parts.'"
('10.3.9.223', 58357): received "b'in parts.'"

socket_server.py

accepted <socket.socket fd=296, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58344)> from ('10.3.9.223', 58344) 1
accepted <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58345)> from ('10.3.9.223', 58345) 1
accepted <socket.socket fd=304, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58346)> from ('10.3.9.223', 58346) 1
accepted <socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58347)> from ('10.3.9.223', 58347) 1
accepted <socket.socket fd=312, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58348)> from ('10.3.9.223', 58348) 1
accepted <socket.socket fd=316, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58349)> from ('10.3.9.223', 58349) 1
accepted <socket.socket fd=320, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58350)> from ('10.3.9.223', 58350) 1
accepted <socket.socket fd=324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58351)> from ('10.3.9.223', 58351) 1
accepted <socket.socket fd=328, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58352)> from ('10.3.9.223', 58352) 1
accepted <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58353)> from ('10.3.9.223', 58353) 1
echoing b'This is the message. ' to <socket.socket fd=296, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58344)>
echoing b'This is the message. ' to <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58345)>
echoing b'This is the message. ' to <socket.socket fd=304, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58346)>
echoing b'This is the message. ' to <socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58347)>
echoing b'This is the message. ' to <socket.socket fd=312, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58348)>
echoing b'This is the message. ' to <socket.socket fd=316, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58349)>
echoing b'This is the message. ' to <socket.socket fd=320, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58350)>
echoing b'This is the message. ' to <socket.socket fd=324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58351)>
echoing b'This is the message. ' to <socket.socket fd=328, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58352)>
echoing b'This is the message. ' to <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58353)>
echoing b'It will be sent ' to <socket.socket fd=296, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58344)>
echoing b'It will be sent ' to <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58345)>
echoing b'It will be sent ' to <socket.socket fd=304, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58346)>
echoing b'It will be sent ' to <socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58347)>
echoing b'It will be sent ' to <socket.socket fd=312, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58348)>
echoing b'It will be sent ' to <socket.socket fd=316, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58349)>
echoing b'It will be sent ' to <socket.socket fd=320, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58350)>
echoing b'It will be sent ' to <socket.socket fd=324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58351)>
echoing b'It will be sent ' to <socket.socket fd=328, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58352)>
echoing b'It will be sent ' to <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58353)>
echoing b'in parts.' to <socket.socket fd=296, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58344)>
echoing b'in parts.' to <socket.socket fd=304, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58346)>
echoing b'in parts.' to <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58345)>
echoing b'in parts.' to <socket.socket fd=320, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58350)>
echoing b'in parts.' to <socket.socket fd=312, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58348)>
echoing b'in parts.' to <socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58347)>
echoing b'in parts.' to <socket.socket fd=316, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58349)>
echoing b'in parts.' to <socket.socket fd=328, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58352)>
echoing b'in parts.' to <socket.socket fd=324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58351)>
echoing b'in parts.' to <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58353)>
closing <socket.socket fd=328, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58352)>
closing <socket.socket fd=320, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58350)>
closing <socket.socket fd=324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58351)>
closing <socket.socket fd=304, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58346)>
closing <socket.socket fd=312, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58348)>
closing <socket.socket fd=308, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58347)>
closing <socket.socket fd=316, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58349)>
closing <socket.socket fd=296, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58344)>
closing <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58345)>
closing <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58353)>
accepted <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58356)> from ('10.3.9.223', 58356) 1
accepted <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58357)> from ('10.3.9.223', 58357) 1
echoing b'This is the message. ' to <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58356)>
echoing b'This is the message. ' to <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58357)>
echoing b'It will be sent ' to <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58356)>
echoing b'It will be sent ' to <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58357)>
echoing b'in parts.' to <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58356)>
echoing b'in parts.' to <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58357)>
closing <socket.socket fd=332, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58356)>
closing <socket.socket fd=300, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('10.3.9.223', 9998), raddr=('10.3.9.223', 58357)>