Production-Ready Beanstalkd with Laravel 4 Queues
原文地址:http://fideloper.com/ubuntu-beanstalkd-and-laravel4
Note: TL;DR version at the bottom!
Queues are a great way to take some task out of the user-flow and put them in the background. Allowing a user to skip waiting for these tasks makes our applications appear faster, and gives us another opportunity to segment our application and business logic out further.
For example, sending emails, deleting accounts and processing images are all potentially long-running or memory-intensive tasks; They make great candidates for work which we can off-load to a queue.
Laravel can accomplish this with its Queue package. Specifically, I use the Beanstalkd work queue with Laravel.
Here's how I set that up to be just about production-ready.
Note: I use Ubuntu for development and often in production. The following is accomplishsed in Ubuntu 12.04 Server LTS. Some instructions may differ for you depending on your OS
Here's what we'll cover:
- Laravel and Queues
- Installing Beanstalkd
- Churning through the queue with Supervisor
Laravel and Queues
Laravel makes using queues very easy. Our application, the "producer"
, can simply run something likeQueue::push('SendEmail', array('message' => $message));
too add a "job"
to the queue.
On the other end of the queue is the code listening for new jobs and a script to process the job (collectively, the "workers"
). This means that in addition to adding jobs to the queue, we need to set up a worker to pull from the stack of available jobs.
Here's how that looks in Laravel. In this example, we'll create an image-processing queue.
Install dependencies
As noted in the docs, Laravel requires the Pheanstalk package for using Beanstalkd. We can install this using Composer:
$ composer require pda/pheanstalk:dev-master
Create a script to process it
Once our PHP dependency in installed, we can begin to write some code. In this example, we'll create aPhotoService
class to handle the processing. If no method is specified, laravel assumes the class will have a fire()
method. This is half of a worker - the code which does some processing.
<?php namespaceMyapp\Queue;classPhotoService{publicfunction fire($job, $data){// Minify, crop, shrink, apply filters or otherwise manipulate the image}}
Push a job to a Queue
When a user uploads an image, we'll add a job to the queue so our worker can process it.
In Laravel, we'll create a job by telling the Queue library what code will handle the job (in this case thefire()
method inside of Myapp\Queue\PhotoService
as defined above) and give it some data to work with. In our example, we simply pass it a path to an image file.
Queue::push('Myapp\Queue\PhotoService', array('image_path'=>'/path/to/image/file.ext'));
Process the jobs
At this point, we have code to process an image (most of a worker), and we've added a job to the queue. The last step is to have code pull a job from the queue.
This is the other half of a worker. The worker needs to both pull a job from the queue and do the processing. In Laravel, that's split into 2 functionalities - Laravel's queue listener, and the code we write ourselves - in this case, the PhotoService
.
Laravel has some CLI tools to help with queues:
// Fire the latest job in the queue
$ php artisan queue:work
// Listen for new jobs in the queue// and fire them off one at a time// as they are created
$ php artisan queue:listen
When not working with the "sync" driver, these tools are what you need to use in order to process the jobs in your queue. We run the queue:listen
command to have laravel listen to the queue and pull jobs as they become available.
Let's install Beanstalkd to see how that works.
By default, laravel will run queue jobs synchronously - that is, it runs the job at the time of creation. This means the image will be processed in the same request that the user created when uploading an image. That's useful for testing, but not for production. We'll make this asynchronous by introducing Beanstalkd.
Beanstalkd
Let's install Beanstalkd:
# Debian / Ubuntu:
$ sudo apt-get update
$ sudo apt-get install beanstalkd
Note: You may be able to get a newer version of Beanstalkd by adding this PPA. Ubuntu 12.04 installs an older version of Beanstalkd.
Next, some quick configuration. The first thing we need to do is tell Beanstalkd to start when the system starts up or reboots. Edit /etc/default/beanstalkd
and set START
to "yes".
$ sudo vim /etc/default/beanstalkd
> START yes # uncomment
Then we can start Beanstalkd:
$ sudo service beanstalkd start
# Alternatively: /etc/init.d/beanstalkd start
Now we can setup Laravel. In your app/config/queue.php
file, set the default queue to 'beanstalkd':
'default'=>'beanstalkd',
Then edit any connection information you need to change. I left my configuration with the defaults as I installed it on the same server as the application.
'connections'=> array('beanstalkd'=> array('driver'=>'beanstalkd','host'=>'localhost','queue'=>'default',),),
Now when we push a job to the queue in Laravel, we'll be pushing to Beanstalkd!
Installing Beanstalkd on a remote server
You may (read: should) want to consider installing Beanstalkd on another server, rather than your application server. Since Beantalkd is an in-memory service, it can eat up your servers resources under load.
To do this, you can install Beanstalkd on another server, and simply point your "host" to the proper server address, rather than localhost
.
This leaves the final detail - what server runs the job? If you follow all other steps here, Supervisord will still be watching Laravel's listener on your application server. You may want to consider running your job script (or even a copy of your application which has a job script) on yet another server whose job is purely to churn through Beanstalkd queue jobs. This means having a listener and working listener/job code on yet another server.
In fact, in a basic distributed setup, we'd probably have an application server (or 2, plus a load-balancer), a database server, a queue server and a job server!
Supervisord
Let's say you pushed a job to Beanstalkd:
Queue::push('Myapp\Queue\PhotoService', array('image_path'=>'/path/to/image/file.ext'));
Now what? You might notice that it goes to Beanstalkd, but Myapp\Queue\PhotoService@fire()
doesn't seem to be getting called. You've checked your error logs, you see if the image was edited, and found that the the job is just "sitting there" in your Beanstalkd queue.
Beanstalkd doesn't actually PUSH jobs to a script - instead, we need a worker to check if there are jobs available and ask for them.
This is what $ php artisan queue:listen
does - It listens for jobs and runs them as they become available.
If you run that command, you'll see your job being sent to code. If all goes well, your image will beproperly manipulated.
The question then becomes: How do I make php listen at all times? We need to avoid having to "supervise" that process manually. This is where Supervisord comes in.
Supervisord will watch our queue:listen
command and restart it if it fails. Let's see how to set that up.
First, we'll install it:
# Debian / Ubuntu:
$ sudo apt-get install supervisor
Next, we'll configure it. We need to define a process to listen to.
$ sudo vim /etc/supervisor/conf.d/myqueue.conf
Add this to your new conf file, changing file paths and your environment as necessary:
[program:myqueue]
command=php artisan queue:listen --env=your_environment
directory=/path/to/laravel
stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log
redirect_stderr=true
We now have a process called "myqueue" which we can tell Supervisord to start and monitor.
Let's do that:
$ sudo supervisorctl
> reread # Tell supervisord to check for new items in /etc/supervisor/conf.d/> add myqueue # Add this process to Supervisord> start myqueue # May say "already started"
Now the "myqueue" process is on and being monitored. If our queue listener fails, Supervisord will restart the php artisan queue:listen --env=your_environment
process.
You can check that it is indeed running that process with this command:
$ ps aux | grep php
# You should see some output like this:
php artisan queue:listen --env=your_environment
sh -c php artisan queue:work --queue="default"--delay=0--memory=128--sleep --env=your_environment
php artisan queue:work --queue=default--delay=0--memory=128--sleep --env=your_environment
Wrapping up
Now we have a full end-to-end queue working and in place!
- We create a script to process a queued job
- We installed Beanstalkd to act as the work queue
- We use Laravel to push jobs to our queue
- We use Laravel
queue:listen
to act as a worker and pull jobs from the queue - We wrote some code to process a job from the queue
- We use Supervisord to ensure
queue:listen
is always listening for new jobs
Notes
- You might want to consider setting up log rotation on the Laravel and Supervisord logs
- You can read here for more information on setting up Supervisord on Ubuntu.
- Read the Laravel docs on queues to learn how and when to
release
ordelete
jobs.
TL;DR
For reference, just copy and paste the whole process from here:
$ sudo apt-get update
$ sudo apt-get install -y beanstalkd supervisor
$ sudo vim /etc/default/beanstalkd
> START yes # uncomment this line
$ sudo service beanstalkd start
$ sudo vim /etc/supervisor/conf.d/myqueue.conf
Enter this, changing as needed:
[program:myqueue]
command=php artisan queue:listen --env=your_environment
directory=/path/to/laravel
stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log
redirect_stderr=true
Start Supervisord:
$ sudo supervisorctl
> reread # Get available jobs> add myqueue
> start myqueue
Read more on Supervisord here for info on supervisorctl.
另外还有一篇相关的不错的文章:http://www.lornajane.net/posts/2014/working-with-php-and-beanstalkd
Production-Ready Beanstalkd with Laravel 4 Queues的更多相关文章
- Practical Node.js (2018版) 第10章:Getting Node.js Apps Production Ready
Getting Node.js Apps Production Ready 部署程序需要知道的方面: Environment variables Express.js in production So ...
- CNCF CloudNative Landscape
cncf landscape CNCF Cloud Native Interactive Landscape 1. App Definition and Development 1. Database ...
- Spring Boot Reference Guide
Spring Boot Reference Guide Authors Phillip Webb, Dave Syer, Josh Long, Stéphane Nicoll, Rob Winch, ...
- 【Kubernetes 系列二】从虚拟机讲到 Kubernetes 架构
目录 什么是虚拟机? 什么是容器? Docker Kubernetes 架构 Kubernetes 对象 基础设施抽象 在认识 Kubernetes 之前,我们需了解下容器,在了解容器之前,我们得先知 ...
- CNCF LandScape Summary
CNCF Cloud Native Interactive Landscape 1. App Definition and Development 1. Database Vitess:itess i ...
- Spring Boot文档
本文来自于springboot官方文档 地址:https://docs.spring.io/spring-boot/docs/current/reference/html/ Spring Boot参考 ...
- Upgrade from SharePoint 2010 to SharePoint 2016
[转]http://nikcharlebois.com/upgrade-from-sharepoint-2010-to-sharepoint-2016/ In this blog, I will go ...
- I finally made sense of front end build tools. You can, too.
来源于:https://medium.freecodecamp.com/making-sense-of-front-end-build-tools-3a1b3a87043b#.nvnd2vsd8 ...
- redis该如何分区-译文(原创)
写在最前,最近一直在研究redis的使用,包括redis应用场景.性能优化.可行性.这是看到redis官网中一个链接,主要是讲解redis数据分区的,既然是官方推荐的,那我就翻译一下,与大家共享. P ...
随机推荐
- WPF判断两个时间大小避免误差
进行查询操作的时候,经常用到判断开始时间和结束时间大小的条件,由于从控件上获取的时间除了年月日时分秒,还包括毫秒.微秒等,导致直接判断时间大小的时候会产生一些误差,如下: 结果分析:年月日时分秒一致的 ...
- nodejs搭建简易的rpc服务
这里主要使用的是jayson包,使用jayson实现rpc server端十分简单,如下: var jayson = require('jayson') // create a server var ...
- linux的日常经常使用的命令
现在经常用到linux命令,又时候回忘记,我就做个小笔记,大家也可以补充补充.....可以评论一下,我会截图做笔记的 netstat -ntlp //查看当前系统进程和端口等信息 tail -f fi ...
- 7、包装类、System、Math、Arrays、大数据运算
基本类型封装 基本数据类型对象包装类概述 *A:基本数据类型对象包装类概述 *a.基本类型包装类的产生 在实际程序使用中,程序界面上用户输入的数据都是以字符串类型进行存储的.而程序开发中,我们需要把字 ...
- 撩课-Java每天5道面试题第9天
撩课Java+系统架构 视频 点击开始学习 76.XML技术的作用? XML技术用于数据存储. 信息配置. 数据交换三方面. 可以将数据存储在XML中, 通过节点. 元素内容. 属性标示数据内容及关系 ...
- fuzhou 1692 Key problem ***
Problem 1692 Key problem Accept: 103 Submit: 553 Time Limit: 1000 mSec Memory Limit : 32768 KB ...
- Redis-SDS
Redis 的简单动态字符串 (simple dynamic string,SDS) SDS的结构: struct sdshdr { int len; //保存的字符串长度. int free; ...
- java 通用对象排序
一个排序类,一个排序util? no.no.no…… 使用反射机制,写了一个通用的对象排序util,欢迎指正. 实体类: package entity; public class BaseTypeEn ...
- 轻松学习java可重入锁(ReentrantLock)的实现原理(转 图解)
前言 相信学过java的人都知道 synchronized 这个关键词,也知道它用于控制多线程对并发资源的安全访问,兴许,你还用过Lock相关的功能,但你可能从来没有想过java中的锁底层的机制是怎么 ...
- JavaScript周报#184
This week’s JavaScript news Read this issue on the Web | Issue Archive JavaScript Weekly Issue 184Ju ...