python10

 一、多进程multiprocessing

multiprocessing包是Python中的多进程管理包。与threading.Thread类似，它可以利用multiprocessing.Process对象来创建一个进程。该Process对象与Thread对象的用法相同，也有start(), run(), join()的方法。

 1 import multiprocessing,threading
 2 import time
 3
 4 def thread_run():
 5     print(threading.get_ident())
 6 def run(name):
 7     time.sleep(2)
 8     print('Hello ',name)
 9     t = threading.Thread(target=thread_run)
10     t.start()
11
12
13 if __name__== '__main__':
14     for i in range(10):
15         p = multiprocessing.Process(target=run, args=('bob_%s'%i,))
16         p.start()

运行结果：

 1 Hello  bob_0
 2 1144
 3 Hello  bob_8
 4 1268
 5 Hello  bob_4
 6 4360
 7 Hello  bob_2
 8 768
 9 Hello  bob_6
10 5308
11 Hello  bob_5
12 Hello  bob_1
13 6076
14 5088
15 Hello  bob_9
16 6104
17 Hello  bob_3
18 5196
19 Hello  bob_7
20 748

二、进程池

如果要创建多个进程，可以使用进程池，启用进程池需要使用Pool库，使用指令pool=Pool()可自动调用所有CPU，

map()函数相当于一个循环，将参数2中的列表元素逐次灌入参数1的函数中。

 1 from multiprocessing import Pool
 2
 3 def squre(num):
 4     return num ** 2
 5
 6
 7 if __name__ == '__main__':
 8     numbers = [0,1,2,3,4,5]
 9     pool = Pool(processes=5)#进程池中最多能放入5个进程
10     print(pool.map(squre,numbers))

运行结果：

[0, 1, 4, 9, 16, 25]

Pool还有以下常用的方法:

• apply_async(func, args)从进程池中取出一个进程执行func，args为func的参数。它将返回一个AsyncResult的对象，我们可以调用get()方法以获得结果。

• close() 关闭进程池，不再创建新的进程

• join() wait()进程池中的全部进程。但是必须对Pool先调用close()方法才能join.

 1 from  multiprocessing import Process, Pool,freeze_support
 2 import time
 3 import os
 4
 5 def Foo(i):
 6     time.sleep(2)
 7     print("in process",os.getpid())
 8     return i + 100
 9
10 def Bar(arg):
11     print('-->exec done:', arg,os.getpid())
12
13 if __name__ == '__main__':
14     #freeze_support()
15     pool = Pool(processes=3)
16     print("主进程",os.getpid())
17     for i in range(10):
18         pool.apply_async(func=Foo, args=(i,), callback=Bar) #callback=回调
19
20     print('end')
21     pool.close()
22     pool.join() 

运行结果：

 1 主进程 5660
 2 end
 3 in process 7048
 4 -->exec done: 100 5660
 5 in process 3396
 6 -->exec done: 101 5660
 7 in process 6728
 8 -->exec done: 102 5660
 9 in process 7048
10 -->exec done: 103 5660
11 in process 3396
12 -->exec done: 104 5660
13 in process 6728
14 -->exec done: 105 5660
15 in process 7048
16 -->exec done: 106 5660
17 in process 3396
18 -->exec done: 107 5660
19 in process 6728
20 -->exec done: 108 5660
21 in process 7048
22 -->exec done: 109 5660

除了主进程，其它结果是三个一组执行的，因为进程池中每次最多有三个进程。

三、进程通信

进程间通信常用两种方法：Queue和pipe,Queue可以用在多个进程间实现通信，pipe用在两个进程间通信。

 1 import os
 2 import multiprocessing
 3 import time
 4 #==================
 5 # input worker
 6 def inputQ(queue):
 7     info = str(os.getpid()) + '(put):' + str(time.time())
 8     queue.put(info)
 9
10 # output worker
11 def outputQ(queue,lock):
12     info = queue.get()
13     lock.acquire()
14     print (str(os.getpid()) + '(get):' + info)
15     lock.release()
16
17 if __name__ == '__main__':
18     record1 = []  # store input processes
19     record2 = []  # store output processes
20     lock = multiprocessing.Lock()  # To prevent messy print
21     queue = multiprocessing.Queue(3)
22
23
24     for i in range(10):
25         process = multiprocessing.Process(target=inputQ, args=(queue,))
26         process.start()
27         record1.append(process)
28
29
30     for i in range(10):
31         process = multiprocessing.Process(target=outputQ, args=(queue, lock))
32         process.start()
33         record2.append(process)
34
35     for p in record1:
36         p.join()
37
38     queue.close()
39
40     for p in record2:
41         p.join()

运行结果：

 1 4004(get):4556(put):1476337412.4875286
 2 512(get):5088(put):1476337412.6345284
 3 8828(get):7828(put):1476337412.7965286
 4 8372(get):1032(put):1476337412.8185284
 5 7740(get):1496(put):1476337412.9205284
 6 4176(get):632(put):1476337412.9855285
 7 5828(get):8508(put):1476337412.9595284
 8 4236(get):9204(put):1476337412.9925284
 9 7632(get):8956(put):1476337413.2055285
10 6376(get):4160(put):1476337413.0705285

Pipe可以是单向(half-duplex)，也可以是双向(duplex)。我们通过mutiprocessing.Pipe(duplex=False)创建单向管道 (默认为双向)。一个进程从PIPE一端输入对象，然后被PIPE另一端的进程接收，单向管道只允许管道一端的进程输入，而双向管道则允许从两端输入。

 1 from multiprocessing import Process, Pipe
 2
 3
 4 def f(conn):
 5     conn.send([42, None, 'hello from child'])
 6     conn.send([42, None, 'hello from child2'])
 7     print("from parent:",conn.recv())
 8     conn.close()
 9
10 if __name__ == '__main__':
11     parent_conn, child_conn = Pipe()
12     p = Process(target=f, args=(child_conn,))
13     p.start()
14     print(parent_conn.recv())  # prints "[42, None, 'hello']"
15     print(parent_conn.recv())  # prints "[42, None, 'hello']"
16     parent_conn.send("chupi可好") # prints "[42, None, 'hello']"
17     p.join()

运行结果：

1 [42, None, 'hello from child']
2 [42, None, 'hello from child2']
3 from parent: chupi可好