celery的安装和使用

celery是python开发的分布式任务调度模块，接口简单，开发容易，五分钟就写出一个异步发送邮件的服务，celery本身不含消息服务，它使用第三方消息服务来传递任务，目前，celery支持的消息服务有RabbitMQ,redis甚至是数据库，redis是最佳选择

Windows使用celery只能安装 3.1.25版

pip install celery==3.1.25  

编写tasks.py

# tasks.py
import time
from celery import Celery

celery = Celery('tasks', broker='redis://localhost:6379/0')

@celery.task
def sendmail(mail):
    print('sending mail to %s...' % mail['to'])
    time.sleep(2.0)
    print('mail sent.')

启动celery处理任务

celery -A tasks worker --loglevel=info

tasks是任务文件名，worker任务角色，--loglevel=info 任务日志级别

上面的命令行实际上启动的是worker，如果要放到后台运行，可以扔给supervisor

要在目录下启动

D:\py3code\jintong_day1\ce>celery -A tasks worker --loglevel=info
[2018-05-30 19:37:15,815: WARNING/MainProcess] d:\py3.6\lib\site-packages\celery
\apps\worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.

  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

 -------------- celery@E73-PC v3.1.25 (Cipater)
---- **** -----
--- * ***  * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x395deb8
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery

[tasks]
  . tasks.sendmail

[2018-05-30 19:37:15,926: INFO/MainProcess] Connected to redis://localhost:6379/
0
[2018-05-30 19:37:15,995: INFO/MainProcess] mingle: searching for neighbors
[2018-05-30 19:37:17,160: INFO/MainProcess] mingle: all alone
[2018-05-30 19:37:17,183: WARNING/MainProcess] celery@E73-PC ready.

怎么发送任务

在当前目录下打开命令行，进入python

>>> from tasks import sendmail
>>> sendmail.delay(dict(to='celery@python.org'))
<AsyncResult: 1a0a9262-7858-4192-9981-b7bf0ea7483b>

在worker里就可以看到任务处理的消息

[2018-05-30 19:36:13,517: INFO/MainProcess] Received task: tasks.sendmail[815178
90-2406-4756-a4b5-d650ea8cd2e2]
[2018-05-30 19:36:13,517: WARNING/Worker-1] sending mail to celery@hhh
[2018-05-30 19:36:15,524: WARNING/Worker-1] mail sent
[2018-05-30 19:36:15,524: INFO/MainProcess] Task tasks.sendmail[81517890-2406-47
56-a4b5-d650ea8cd2e2] succeeded in 1.9970000000030268s: None

celery默认设置就能满足基本要求，worker以pool模式启动，默认大小为CPU核心数量，缺省化机制为pickle,但可以指定为json,由于python调用UNIX/Linux太容易，所以，用celery作为异步任务框架非常合适

celery还有一些高级用法，比如把多个任务组合成一个原子任务，还有一个完善的监控接口