What is Gearman?

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates. A few strong points about Gearman:

  • Open Source It’s free! (in both meanings of the word) Gearman has an active open source community that is easy to get involved with if you need help or want to contribute. Worried about licensing? Gearman is BSD.
  • Multi-language - There are interfaces for a number of languages, and this list is growing. You also have the option to write heterogeneous applications with clients submitting work in one language and workers performing that work in another.
  • Flexible - You are not tied to any specific design pattern. You can quickly put together distributed applications using any model you choose, one of those options being Map/Reduce.
  • Fast - Gearman has a simple protocol and interface with an optimized, and threaded, server written in C/C++ to minimize your application overhead.
  • Embeddable - Since Gearman is fast and lightweight, it is great for applications of all sizes. It is also easy to introduce into existing applications with minimal overhead.
  • No single point of failure - Gearman can not only help scale systems, but can do it in a fault tolerant way.
  • No limits on message size - Gearman supports single messages up to 4gig in size. Need to do something bigger? No problem Gearman can chunk messages.
  • Worried about scaling? - Don’t worry about it with Gearman. Craig’s List, Tumblr, Yelp, Etsy,… discover what others have known for years.

Content is being updated regularly, so please check back often. You may also want to check out other forms of communication if you would like to learn more or get involved!

How Does Gearman Work?

A Gearman powered application consists of three parts: a client, a worker, and a job server. The client is responsible for creating a job to be run and sending it to a job server. The job server will find a suitable worker that can run the job and forwards the job on. The worker performs the work requested by the client and sends a response to the client through the job server. Gearman provides client and worker APIs that your applications call to talk with the Gearman job server (also known as gearmand) so you don’t need to deal with networking or mapping of jobs. Internally, the gearman client and worker APIs communicate with the job server using TCP sockets. To explain how Gearman works in more detail, lets look at a simple application that will reverse the order of characters in a string. The example is given in PHP, although other APIs will look quite similar.

We start off by writing a client application that is responsible for sending off the job and waiting for the result so it can print it out. It does this by using the Gearman client API to send some data associated with a function name, in this case the function reverse. The code for this is (with error handling omitted for brevity):

<?php
// Reverse Client Code
$client = new GearmanClient();
$client->addServer();
print $client->doNormal("reverse", "Hello World!");

This code initializes a client class, configures it to use a job server with add_server (no arguments means use 127.0.0.1 with the default port), and then tells the client API to run the reverse function with the workload “Hello world!”. The function name and arguments are completely arbitrary as far as Gearman is concerned, so you could send any data structure that is appropriate for your application (text or binary). At this point the Gearman client API will package up the job into a Gearman protocol packet and send it to the job server to find an appropriate worker that can run the reverse function. Let’s now look at the worker code:

<?php
// Reverse Worker Code
$worker = new GearmanWorker();
$worker->addServer();
$worker->addFunction("reverse", function ($job) {
return strrev($job->workload());
});
while ($worker->work());

This code defines a function my_reverse_function that takes a string and returns the reverse of that string. It is used by a worker object to register a function named reverse after it is setup to connect to the same local job server as the client. When the job server receives the job to be run, it looks at the list of workers who have registered the function name reverse and forwards the job on to one of the free workers. The Gearman worker API then takes this request, runs the function my_reverse_function, and sends the result of that function back through the job server to the client.

As you can see, the client and worker APIs (along with the job server) deal with the job management and network communication so you can focus on the application parts. There a few different ways you can run jobs in Gearman, including background for asynchronous processing and prioritized jobs. See the documentation available for the various APIs for details.

How Is Gearman Useful?

The reverse example above seems like a lot of work to run a function, but there are a number of ways this can be useful. The simplest answer is that you can use Gearman as an interface between a client and a worker written in different languages. If you want your PHP web application to call a function written in C, you could use the PHP client API with the C worker API, and stick a job server in the middle. Of course, there are more efficient ways of doing this (like writing a PHP extension in C), but you may want a PHP client and a Python worker, or perhaps a MySQL client and a Perl worker. You can mix and match any of the supported language interfaces easily, you just need all applications to be able to understand the workload being sent. Is your favorite language not supported yet? Get involved with the project, it’s probably fairly easy for either you or one of the existing Gearman developers to put a language wrapper on top of the C library.

The next way that Gearman can be useful is to put the worker code on a separate machine (or cluster of machines) that are better suited to do the work. Say your PHP web application wants to do image conversion, but this is too much processing to run it on the web server machines. You could instead ship the image off to a separate set of worker machines to do the conversion, this way the load does not impact the performance of your web server and other PHP scripts. By doing this, you also get a natural form of load balancing since the job server only sends new jobs to idle workers. If all the workers running on a given machine are busy, you don’t need to worry about new jobs being sent there. This makes scale-out with multi-core servers quite simple: do you have 16 cores on a worker machine? Start up 16 instances of your worker (or perhaps more if they are not CPU bound). It is also seamless to add new machines to expand your worker pool, just boot them up, install the worker code, and have them connect to the existing job server.

Now you’re probably asking what if the job server dies? You are able to run multiple job servers and have the clients and workers connect to the first available job server they are configured with. This way if one job server dies, clients and workers automatically fail over to another job server. You probably don’t want to run too many job servers, but having two or three is a good idea for redundancy. The diagram to the left shows how a simple Gearman cluster may look.

From here, you can scale out your clients and workers as needed. The job servers can easily handle having hundreds of clients and workers connected at once. You can draw your own physical (or virtual) machine lines where capacity allows, potentially distributing load to any number of machines. For more details on specific uses and installations, see the section on examples.

支持多语言,client可以为php, worker可以为任何的语言;

Gearman 分布式的异步任务分发框架的更多相关文章

  1. 分布式的任务分发框架-Gearman

    官方文档:http://gearman.org/getting-started/ 安装方法和示例都有,可以详细看一下. Gearman是一个分发任务的程序框架,可以用在各种场合,与Hadoop相比,G ...

  2. 分布式任务分发框架Gearman教程和PHP实现实例

    1.Gearman介绍和使用场景 Gearman是一个分发任务的程序框架,可以用在各种场合,与Hadoop相 比,Gearman更偏向于任务分发功能.它的任务分布非常简单,简单得可以只需要用脚本即可完 ...

  3. Microsoft Orleans构建高并发、分布式的大型应用程序框架

    Microsoft Orleans 在.net用简单方法构建高并发.分布式的大型应用程序框架. 原文:http://dotnet.github.io/orleans/ 在线文档:http://dotn ...

  4. 分布式事务最终一致性-CAP框架轻松搞定

    前言 对于分布式事务,常用的解决方案根据一致性的程度可以进行如下划分: 强一致性(2PC.3PC):数据库层面的实现,通过锁定资源,牺牲可用性,保证数据的强一致性,效率相对比较低. 弱一致性(TCC) ...

  5. android 异步加载框架 原理完全解析

    一.手写异步加载框架MyAsycnTask(核心原理) 1.我为大家手写了一个异步加载框架,涵盖了异步加载框架核心原理. MyAsycnTask.java import android.os.Hand ...

  6. Javascript:自己写异步流程编程框架

    背景 自从NodeJs出来以后,异步编程便更加系统化和框架话了,为了应对异步编程框架带来的深层嵌套问题,社区也出现了异步流程编程框架,本文主要对异步流程框架做一个简单的解析. 现配代码了 var As ...

  7. 可扩展多线程异步Socket服务器框架EMTASS 2.0 (转自:http://blog.csdn.net/hulihui)

    可扩展多线程异步Socket服务器框架EMTASS 2.0 (转自:http://blog.csdn.net/hulihui) 0 前言 >>[前言].[第1节].[第2节].[第3节]. ...

  8. 异步处理的框架Sanic的使用方法和小技巧

    Sanic是异步处理的框架,运用Sanic可以开发快速异步响应的web程序.想必大家看到这个都会比较期待和兴奋. 那么如何使用Sanic来实现快速响应呢?我们先来看一看Sanic的基本介绍. Sani ...

  9. 异步任务分发模块Celery

    Celery简介 Celery是一个功能完备即插即用的任务队列.它使得我们不需要考虑复杂的问题,使用非常简单. celery适用异步处理问题,当遇到发送邮件.或者文件上传, 图像处理等等一些比较耗时的 ...

随机推荐

  1. Mybatis 别名机制,自动扫描 数据的增删改

    mybatis别名机制: 在mybatis.xml文件中的<configuration></configuration>标签中间加入属性<typeAliases>& ...

  2. 【Oracle】实现Oracle数据库对象的一键升级

    引言     公司内部的项目比较倾向于将业务逻辑放在oracle存储过程中实现,所以每次项目升级都涉及到很多的oracle表,存储过程等数据库对象的升级.然而采取的升级方式是比较"原始&qu ...

  3. XSLT 教程

    应用自http://www.w3school.com.cn/xsl/index.asp XSL 指扩展样式表语言(EXtensible Stylesheet Language). 万维网联盟开始发展 ...

  4. spring学习-5

    spring表达式SpEL 语法#{..},为bean的属性进行动态赋值 通过bean的id对bean进行引用 调用方法以及引用对象中的属性 计算表达式的值 正则表达式的匹配 修改Address.ja ...

  5. 用Java实现异构数据库的高效通用分页查询功能

    不同数据库的分页查询语句有着较大区别,其中MySQL数据的limit offset语法最为简单,而SQL Server数据库和Oracle数据库的分页就比较复杂了. 网上常见的SQL Server和O ...

  6. 关于C语言字符串函数使用的一点心得

    就字符串的拼接函数为例strcat. 原型:extern char *strcat(char *dest,char *src);用法:#include <string.h> 功能:把src ...

  7. 为什么很多公司招聘前端开发要求有 Linux / Unix 下的开发经验?

    知乎: http://www.zhihu.com/question/19666395

  8. 异常:java.lang.IllegalStateException: No instances found of configserver(里面是一个微服务名)

    今天本地测试代码时出现了个异常,该异常出现的原因是:微服务启动的顺序出现了问题: 应该先启动本地eureka,然后在启动本地配置中心,然后在启动具体的微服务.

  9. Filebeat 5.x 日志收集器 安装和配置

    Filebeat 5.x版本 风来了.fox 1.下载和安装 https://www.elastic.co/downloads/beats/filebeat 这里选择 LINUX 64-BIT 即方式 ...

  10. 最小LINUX系统下U盘的挂载及卸载

    U盘挂载命令U盘插入的时候会显示启动信息,启动信息中sda: sda1指U盘的设备名为sda1dev设备目录下有一个sda1设备文件,此设备文件就是我们插入的U盘,我们将这个设备文件挂载到Linux系 ...