python 多进程通讯三种方法性能对比(queue, pipe, zeromq)

当然，这三种办法都会在两个进程之间把数据复制一遍，效率肯定没有 shared memory 高，但是考虑到这三种方式都不用考虑锁之类东西，用起来是比较方便的。这三种方式的实现的功能都是差不多的，但是在不同的情境下它们各自的性能怎么样呢？

下面给出测试的代码，逻辑都是差不多的。本次的测试数据是一个长度为size大小的纯int列表

这里测试两组数据，一组是“少量多次”size=5,times=5000, 一组是“多量少次”size=8000,times=45

pipe 测试代码

from multiprocessing import Process, Queue, Pipe
import time, os
import random

size = int(input("data size:"))
times = int(input("test times:"))

test_data = [[random.randint(0,1000) for i in range(size)] for j in range(times)] 

def worker(que):
    st = time.time()
    for i in range(times):
        que.recv()
    print(f"time: {time.time() - st}")

conn1, conn2 = Pipe()
# queue =  Queue()
Process(target=worker, args=(conn2,)).start()

for i in range(times):
    conn1.send(test_data)

queue 测试代码

from multiprocessing import Process, Queue, Pipe
import time, os
import random

size = int(input("data size:"))
times = int(input("test times:"))

test_data = [[random.randint(0,1000) for i in range(size)] for j in range(times)]

def worker(que:Queue):
    st = time.time()
    for i in range(times):
        que.get()
    print(f"time: {time.time() - st}")

# conn1, conn2 = Pipe()
queue =  Queue()
Process(target=worker, args=(queue,)).start()

for i in range(times):
    queue.put(test_data)

ZeroMQ 测试代码

from multiprocessing import Process, Queue, Pipe
import time, os
import zmq
import pickle
import random

context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.bind("tcp://*:5557")

size = int(input("data size:"))
times = int(input("test times:"))

test_data = [[random.randint(0,1000) for i in range(size)] for j in range(times)] 

def worker():
    it = time.time()

    context = zmq.Context()
    recive = context.socket(zmq.PULL)
    recive.connect('tcp://127.0.0.1:5557')

    print(f"init time: {time.time() - it}")

    st = time.time()
    for i in range(times):
        pickle.loads(recive.recv())
    print(f"time: {time.time() - st}")

Process(target=worker).start()

for i in range(times):
    socket.send(pickle.dumps(test_data))

（ZeroMQ 这里额外说明一下，因为 ZeroMQ 不能直接传输列表，只能序列化后传输，为了模拟真实情境我就把序列化的时间也算进去了）

结果：

本来想列个表对比，现在看来差距不大，我还测了别的数据，感觉浮动比较大 (当然，这个实验是不严谨的)