笔记-python-standard library-17.7 queue

1. queue

source code:Lib/queue.py

该模块实现了多生产者，多消费者队列。

此模块实现了所有的required locking semantics.

模块支持三种类型的队列，区别仅仅在于检索的顺序。

三种队列分别是FIFO,LIFO，优先级队列（使用heaq模块，优先抛出最小值）。

1.1. 模块中定义的类

class queue.Queue(maxsize=0)

class queue.LifoQueue(maxsize=0)

class queue.PriorityQueue(maxsize=0)

注意：优先级队列中优先抛出最小值。

exception queue.Empty

Exception raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.

exception queue.Full

Exception raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.

1.2. 类方法

三种队列类都提供了以下方法：

1. Queue.qsize()返回队列大致的大小，注意qsize()>0不代表get不会阻塞，同样，qsize()<maxsize不代表put不会阻塞

2. Queue.empty()

3. Queue.full()

4. Queue.put(item, block=True, timeout=None)

block决定是否阻塞，timeout决定最长阻塞时间

5. Queue.put_nowait(item)

6. Queue.get_nowait()： equal to get(False)

下面两个方法来跟踪队列中的任务是否被执行完。

7. Queue.task_done()

8. Queue.join()

阻塞直到队列中的所有任务完成。

其它属性

maxsize：队列最大长度，可以在初始化时给出，也可创建后手动设定。

1.1. productor and consumer

queue实现生产者与消费者

q = queue.Queue()

def productor(arg):
    while True:
        if q.qsize()
< 30:
            q.put(str(arg)+'banana')
        time.sleep(1)

def consumer(arg):
    while True:
        print('con {},
pro {}'.format(arg, q.get()))
        time.sleep(2)

for i in range(3):
t = threading.Thread(target=productor,
args=(i,))
t.start()

for j in range(5):
t = threading.Thread(target=consumer,
args=(j,))
t.start()

1.2. task_done and join

文档如下：

def task_done(self):

'''Indicate that a formerly enqueued task is complete.

Used by Queue consumer threads.
For each get() used to fetch a task,

a subsequent call to task_done() tells the queue that the processing

on the task is complete.

If a join() is currently blocking, it will resume when all items

have been processed (meaning that a task_done() call was received

for every item that had been put() into the queue).

Raises a ValueError if called more times than there were items

placed in the queue.

'''

with self.all_tasks_done:

unfinished = self.unfinished_tasks - 1

if unfinished <= 0:

if unfinished < 0:

raise
ValueError('task_done() called too many times')

self.all_tasks_done.notify_all()

self.unfinished_tasks = unfinished

关键是如果不使用task_done，阻塞不会结束，

下面的代码把红色行删掉，线程会阻塞在q.join()处，

q = queue.Queue()
q.maxsize = 100

def productor(arg):
    while True:
        if q.qsize()
>50:
            q.join()
        else:
            q.put(str(arg)+' banana')
        time.sleep(0.5)

def consumer(arg):
    while True:
        print('con {},
pro {}'.format(arg, q.get()))
        q.task_done()
        time.sleep(2)

def start_t():
    for i in range(4):
        t = threading.Thread(target=productor,
args=(i,))
        t.start()

for j in range(5):
t = threading.Thread(target=consumer,
args=(j,))
t.start()

time.sleep(1)
    while True:
        q_length = q.qsize()
        if q_length
== 0:
            pass
            #break
        else:
            print("queue's
size is {}".format(q_length))
            time.sleep(2)

start_t()
time.sleep(0.5)
print(r'end {}')
print(threading.enumerate())

2.
小结：

queue是线程安全的
FIFO实际使用的是dqueue，
在使用join时一定要使用task_done