python multiprocess不能完全关闭socket的验证

近日项目有原来的多线程升级成为多进程模型后，但出现了个问题，在持续运行一天左右系统处理能力开始变慢，并不时打印以下信息：

too many opened files

修改ulimit中open files为10240之后,运行时间稍微变长，但还是会出现该问题。

使用iostat查看统计信息没发现异常，使用netstat 发现系统连接信息如下

#netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
TIME_WAIT 59
CLOSE_WAIT 28199
ESTABLISHED 170

　　出现了大量的CLOSE_WAIT，应该是此处导致了fd泄露，抓包分析，结果如下

1 11:22:11.613144 IP 127.0.0.1.55313 > 127.0.0.1.9000: Flags [S], seq 1383947720, win 32792, options [mss 16396,sackOK,TS val 2912656137 ecr 0,nop,wscale 7], length 0
2 11:22:11.613175 IP 127.0.0.1.9000 > 127.0.0.1.55313: Flags [S.], seq 3913709618, ack 1383947721, win 32768, options [mss 16396,sackOK,TS val 2912656137 ecr 2912656137,nop,wscale 7], length 0
3 11:22:11.613192 IP 127.0.0.1.55313 > 127.0.0.1.9000: Flags [.], ack 1, win 257, options [nop,nop,TS val 2912656137 ecr 2912656137], length 0
4 11:22:11.613435 IP 127.0.0.1.55313 > 127.0.0.1.9000: Flags [P.], seq 1:10, ack 1, win 257, options [nop,nop,TS val 2912656137 ecr 2912656137], length 9
5 11:22:11.613446 IP 127.0.0.1.9000 > 127.0.0.1.55313: Flags [.], ack 10, win 256, options [nop,nop,TS val 2912656137 ecr 2912656137], length 0
6 11:22:11.632594 IP 127.0.0.1.9000 > 127.0.0.1.55313: Flags [P.], seq 1:10, ack 10, win 256, options [nop,nop,TS val 2912656156 ecr 2912656137], length 9
7 11:22:11.632650 IP 127.0.0.1.55313 > 127.0.0.1.9000: Flags [.], ack 10, win 257, options [nop,nop,TS val 2912656156 ecr 2912656156], length 0
8 11:22:11.632732 IP 127.0.0.1.55313 > 127.0.0.1.9000: Flags [F.], seq 10, ack 10, win 257, options [nop,nop,TS val 2912656156 ecr 2912656156], length 0
9 11:22:11.672056 IP 127.0.0.1.9000 > 127.0.0.1.55313: Flags [.], ack 11, win 256, options [nop,nop,TS val 2912656196 ecr 2912656156], length 0

可以看出，在该次第通信中，前期数据包交互一切正常，但在连接结束之后client端发送了fin报文，而server没有发送Fin包，导致大量的tcp连接停留在close_wai状态，close_wait的分析参考这里http://blog.chinaunix.net/uid-9688646-id-3469570.html

追查代码是很确定调用了close函数关闭socket

因为之前多线程时候没有类似的问题，所以考虑是不是引入多进程之后导致的问题

为此编写多进程版本的socket模型，查看其数据包交互模型，参考网上一哥们的代码，如下

 1 #multiprocessserver.py
 2 import multiprocessing
 3 import socket
 4
 5 def handle(connection, address):
 6     import logging
 7     logging.basicConfig(level=logging.DEBUG)
 8     logger = logging.getLogger("process-%r" % (address,))
 9     try:
10         logger.debug("Connected %r at %r", connection, address)
11         while True:
12             data = connection.recv(1024)
13             if data == "":
14                 logger.debug("Socket closed remotely")
15                 break
16             logger.debug("Received data %r", data)
17             connection.sendall(data)
18             logger.debug("Sent data")
19     except:
20         logger.exception("Problem handling request")
21     finally:
22         logger.debug("Closing socket")
23         connection.close()
24
25 class Server(object):
26     def __init__(self, hostname, port):
27         import logging
28         self.logger = logging.getLogger("server")
29         self.hostname = hostname
30         self.port = port
31
32     def start(self):
33         self.logger.debug("listening")
34         self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
35         self.socket.bind((self.hostname, self.port))
36         self.socket.listen(1)
37
38         while True:
39             conn, address = self.socket.accept()
40             self.logger.debug("Got connection")
41             process = multiprocessing.Process(target=handle, args=(conn, address))
42             process.daemon = True
43             process.start()
44             self.logger.debug("Started process %r", process)
45
46 if __name__ == "__main__":
47     import logging
48     logging.basicConfig(level=logging.DEBUG)
49     server = Server("0.0.0.0", 9000)
50     try:
51         logging.info("Listening")
52         server.start()
53     except:
54         logging.exception("Unexpected exception")
55     finally:
56         logging.info("Shutting down")
57         for process in multiprocessing.active_children():
58             logging.info("Shutting down process %r", process)
59             process.terminate()
60             process.join()
61     logging.info("All done")

 1 #mulclient.py
 2 import socket
 3
 4 if __name__ == "__main__":
 5     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 6     sock.connect(("localhost", 9000))
 7     data = "some data"
 8     sock.sendall(data)
 9     result = sock.recv(1024)
10     print result
11     sock.close()

验证了上面的假设

结论：

python2.6.6版本的多进程multiprocessing在关闭socket时是有问题的，会导致close函数无效从而引发大量close_wait状态的tcp链接