from: http://agiliq.com/blog/2013/10/producer-consumer-problem-in-python/

By : Akshar Raaj

We will solve Producer Consumer problem in Python using Python threads. This problem is nowhere as hard as they make it sound in colleges.

This blog will make more sense if you have some idea about Producer Consumer problem.

Why care about Producer Consumer problem:

  • Will help you understand more about concurrency and different concepts of concurrency.
  • The concept of Producer Consumer problem is used to some extent in implementing a message queue. And you will surely need message queue at some point of time.

While we use threads, you will learn about the following thread topics:

  • Condition in threads.
  • wait() method available on Condition instances.
  • notify() method available on Condition instances.

I will assume you are comfortable with basics of Threads, race condition and how to prevent race condition i.e using locks. If not, my last post on basics of Threadsshould be able to help.

Quoting Wikipedia:

The producer's job is to generate a piece of data, put it into the buffer and start again.
At the same time, the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time

The catch here is "At the same time". So, producer and consumer need to run concurrently. Hence we need separate threads for Producer and Consumer.

from threading import Thread

class ProducerThread(Thread):
def run(self):
pass class ConsumerThread(Thread):
def run(self):
pass

Quoting Wikipedia again:

The problem describes two processes, the producer and the consumer, who share a common,
fixed-size buffer used as a queue.

So we keep one variable which will be global and will be modified by both Producer and Consumer threads. Producer produces data and adds it to the queue. Consumer consumes data from the queue i.e removes it from the queue.

queue = []

In first iteration, we will not put fixed-size constraint on queue. We will make it fixed-size once our basic program works.

Initial buggy program:

from threading import Thread, Lock
import time
import random queue = []
lock = Lock() class ProducerThread(Thread):
def run(self):
nums = range(5) #Will create the list [0, 1, 2, 3, 4]
global queue
while True:
num = random.choice(nums) #Selects a random number from list [0, 1, 2, 3, 4]
lock.acquire()
queue.append(num)
print "Produced", num
lock.release()
time.sleep(random.random()) class ConsumerThread(Thread):
def run(self):
global queue
while True:
lock.acquire()
if not queue:
print "Nothing in queue, but consumer will try to consume"
num = queue.pop(0)
print "Consumed", num
lock.release()
time.sleep(random.random()) ProducerThread().start()
ConsumerThread().start()

Run it few times and notice the result. Your program might not end after raising IndexError. Use Ctrl+Z to terminate.

Sample output:

Produced 3
Consumed 3
Produced 4
Consumed 4
Produced 1
Consumed 1
Nothing in queue, but consumer will try to consume
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "producer_consumer.py", line 31, in run
num = queue.pop(0)
IndexError: pop from empty list

Explanation:

  • We started one producer thread(hereafter referred as producer) and one consumer thread(hereafter referred as consumer).
  • Producer keeps on adding to the queue and consumer keeps on removing from the queue.
  • Since queue is a shared variable, we keep it inside lock to avoid race condition.
  • At some point, consumer has consumed everything and producer is still sleeping. Consumer tries to consume more but since queue is empty, an IndexError is raised.
  • But on every execution, before IndexError is raised you will see the print statement telling "Nothing in queue, but consumer will try to consume", which explains why you are getting the error.

We found this implementaion as the wrong behaviour.

What is the correct behaviour?

When there was nothing in the queue, consumer should have stopped running and waited instead of trying to consume from the queue. And once producer adds something to the queue, there should be a way for it to notify the consumer telling it has added something to queue. So, consumer can again consume from the queue. And thus IndexError will never be raised.

About Condition

  • Condition object allows one or more threads to wait until notified by another thread. Taken from here.

And this is exactly what we want. We want consumer to wait when the queue is empty and resume only when it gets notified by the producer. Producer should notify only after it adds something to the queue. So after notification from producer, we can be sure that queue is not empty and hence no error can crop if consumer consumes.

  • Condition is always associated with a lock.
  • A condition has acquire() and release() methods that call the corresponding methods of the associated lock.

Condition provides acquire() and release() which calls lock's acquire() and release() internally, and so we can replace lock instances with condition instances and our lock behaviour will keep working properly.

Consumer needs to wait using a condition instance and producer needs to notify the consumer using the condition instance too. So, they must use the same condition instance for the wait and notify functionality to work properly.

Let's rewrite our Consumer and Producer code:

from threading import Condition

condition = Condition()

class ConsumerThread(Thread):
def run(self):
global queue
while True:
condition.acquire()
if not queue:
print "Nothing in queue, consumer is waiting"
condition.wait()
print "Producer added something to queue and notified the consumer"
num = queue.pop(0)
print "Consumed", num
condition.release()
time.sleep(random.random())

Let's rewrite Producer code:

class ProducerThread(Thread):
def run(self):
nums = range(5)
global queue
while True:
condition.acquire()
num = random.choice(nums)
queue.append(num)
print "Produced", num
condition.notify()
condition.release()
time.sleep(random.random())

Sample output:

Produced 3
Consumed 3
Produced 1
Consumed 1
Produced 4
Consumed 4
Produced 3
Consumed 3
Nothing in queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
Consumed 2
Nothing in queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
Consumed 2
Nothing in queue, consumer is waiting
Produced 3
Producer added something to queue and notified the consumer
Consumed 3
Produced 4
Consumed 4
Produced 1
Consumed 1

Explanation:

  • For consumer, we check if the queue is empty before consuming.
  • If yes then call wait() on condition instance.
  • wait() blocks the consumer and also releases the lock associated with the condition. This lock was held by consumer, so basically consumer loses hold of the lock.
  • Now unless consumer is notified, it will not run.
  • Producer can acquire the lock because lock was released by consumer.
  • Producer puts data in queue and calls notify() on the condition instance.
  • Once notify() call is made on condition, consumer wakes up. But waking up doesn't mean it starts executing.
  • notify() does not release the lock. Even after notify(), lock is still held by producer.
  • Producer explicitly releases the lock by using condition.release().
  • And consumer starts running again. Now it will find data in queue and no IndexError will be raised.

Adding a max size on the queue

Producer should not put data in the queue if the queue is full.

It can be accomplished in the following way:

  • Before putting data in queue, producer should check if the queue is full.
  • If not, producer can continue as usual.
  • If the queue is full, producer must wait. So call wait() on condition instance to accomplish this.
  • This gives a chance to consumer to run. Consumer will consume data from queue which will create space in queue.
  • And then consumer should notify the producer.
  • Once consumer releases the lock, producer can acquire the lock and can add data to queue.

Final program looks like:

from threading import Thread, Condition
import time
import random queue = []
MAX_NUM = 10
condition = Condition() class ProducerThread(Thread):
def run(self):
nums = range(5)
global queue
while True:
condition.acquire()
if len(queue) == MAX_NUM:
print "Queue full, producer is waiting"
condition.wait()
print "Space in queue, Consumer notified the producer"
num = random.choice(nums)
queue.append(num)
print "Produced", num
condition.notify()
condition.release()
time.sleep(random.random()) class ConsumerThread(Thread):
def run(self):
global queue
while True:
condition.acquire()
if not queue:
print "Nothing in queue, consumer is waiting"
condition.wait()
print "Producer added something to queue and notified the consumer"
num = queue.pop(0)
print "Consumed", num
condition.notify()
condition.release()
time.sleep(random.random()) ProducerThread().start()
ConsumerThread().start()
Sample output:
Produced 0
Consumed 0
Produced 0
Produced 4
Consumed 0
Consumed 4
Nothing in queue, consumer is waiting
Produced 4
Producer added something to queue and notified the consumer
Consumed 4
Produced 3
Produced 2
Consumed 3

Update:

Many people on the internet suggested that I use Queue.Queue instead of using a list with conditions and lock. I agree, but I wanted to show how Conditions, wait() and notify() work so I took this approach.

Let's update our code to use Queue.

Queue encapsulates the behaviour of Condition, wait(), notify(), acquire() etc.

Now is a good time to read the documentation for Queue and the source code for it.

Updated program:

from threading import Thread
import time
import random
from Queue import Queue queue = Queue(10) class ProducerThread(Thread):
def run(self):
nums = range(5)
global queue
while True:
num = random.choice(nums)
queue.put(num)
print "Produced", num
time.sleep(random.random()) class ConsumerThread(Thread):
def run(self):
global queue
while True:
num = queue.get()
queue.task_done()
print "Consumed", num
time.sleep(random.random()) ProducerThread().start()
ConsumerThread().start()
Explanation
  • In place of list, we are using a Queue instance(hereafter queue).
  • queue has a Condition and that condition has its lock. You don't need to bother about Condition and Lock if you use Queue.
  • Producer uses put available on queue to insert data in the queue.
  • put() has the logic to acquire the lock before inserting data in queue.
  • Also put() checks whether the queue is full. If yes, then it calls wait()internally and so producer starts waiting.
  • Consumer uses get.
  • get() acquires the lock before removing data from queue.
  • get() checks if the queue is empty. If yes, it puts consumer in waiting state.
  • get() and put() has proper logic for notify() too. Why don't you check the source code for Queue now?

Producer-consumer problem in Python的更多相关文章

  1. .net IO异步和Producer/Consumer队列实现一分钟n次http请求

    简介 最近工作中有一个需求:要求发送http请求到某站点获取相应的数据,但对方网站限制了请求的次数:一分钟最多200次请求. 搜索之后,在stackoverflow网站查到一个类似的问题..但里面用到 ...

  2. C# Producer Consumer (生产者消费者模式)demo

    第一套代码将producer Consumer的逻辑写到from类里了,方便在demo的显示界面动态显示模拟生产和消费的过程.     第二套代码将producer Consumer的逻辑单独写到一个 ...

  3. Kafka 学习笔记之 Kafka0.11之producer/consumer(Scala)

    Kafka0.11之producer/consumer(Scala): KafkaConsumer: import java.util.Properties import org.apache.kaf ...

  4. Python实现:生产者消费者模型(Producer Consumer Model)

    #!/usr/bin/env python #encoding:utf8 from Queue import Queue import random,threading,time #生产者类 clas ...

  5. Kafka Producer Consumer

    Producer API org.apache.kafka.clients.producer.KafkaProducer props.put("bootstrap.servers" ...

  6. asyncio标准库7 Producer/consumer

    使用asyncio.Queue import asyncio import random async def produce(queue, n): for x in range(1, n + 1): ...

  7. Producer & Consumer

    需要与Eureka结合使用 Eureka环境搭建 Producer 一.pom文件 <?xml version="1.0" encoding="UTF-8" ...

  8. RocketMQ学习笔记(6)----RocketMQ的Client的使用 Producer/Consumer

    1.  添加依赖 pom.xml如下: <dependency> <groupId>org.apache.rocketmq</groupId> <artifa ...

  9. Kafka 学习笔记之 Producer/Consumer (Scala)

    既然Kafka使用Scala写的,最近也在慢慢学习Scala的语法,虽然还比较生疏,但是还是想尝试下用Scala实现Producer和Consumer,并且用HashPartitioner实现消息根据 ...

随机推荐

  1. centos7 添加用户并授权root权限

    centos7.2 版本,我们不可能每次都用root账号登陆,root账号公认的不安全,所以,首先,禁止掉root账号的登陆.1.静止root账号登陆. $ vi /etc/ssh/sshd.conf ...

  2. Centos7 通过SSH使用密钥实现免密登录

    Public Key认证的主要魅力在于认证时承诺不必提供密码就能够同远程系统建立连接. Public Key认证的基础在于一对密钥,public key和private key,public key对 ...

  3. 特殊字符搜索网站 http://symbolhound.com/

    最近在学习makefile,想搜索一下 $@是啥意思,结果google由于忽略了特殊字符,结果啥也没找到, 后来在stackoverflow上看到了别人同样的问题 http://stackoverfl ...

  4. PHP安全之register_globals (转)

    一.register_globals = Off 和 register_globals = On的区别 register_globals是php.ini里的一个配置,这个配置影响到php如何接收传递过 ...

  5. 深入理解ASP.NET MVC(1)

    系列目录 ASP.NET MVC请求的服务过程 下图是书中的截图,表述了一次通常的ASP.NET MVC请求的服务过程: 从图中可以初步看出一个HttpRequest是如何被ASP.NET和ASP.N ...

  6. 3台服务器Redis高可用哨兵模式实现(转)

    http://www.linuxidc.com/Linux/2017-05/143521.htm

  7. 关于 ake sure class name exists, is public, and has an empty constructor that is public

    解决方法:自定义的fragment最好有一个Public的参数为空的构造函数,若需要传入一个参数,可以使用下面的方法 public FileViewFragment(){ } public stati ...

  8. Python基础学习笔记,进阶学习笔记 出处

    参考 虫师-博客 http://www.cnblogs.com/fnng/category 随笔分类相关文章

  9. Air test ios类使用

    1.iOS手机的滑动 相关代码 #python class IOS(Device): ... @property #获取屏幕的尺寸 def display_info(self): if not sel ...

  10. 关于Linux服务器磁盘空间占满问题的解决方法

    下面给大家分享一篇关于Linux服务器磁盘占满问题解决方法(/dev/sda3 满了),需要的的朋友参考下吧   下面我们一起来看一篇关于Linux服务器磁盘占满问题解决(/dev/sda3 满了), ...