安装

pip install celery  #

pip install flower #celery的及时监控组件

配置

默认celery的配置文件名是celeryconfig.py,本例的内容如下：

BROKER_URL ='amqp://guest:guest[@localhost](https://my.oschina.net/u/570656):5672' 
CELERY_ENABLE_UTC = True 
CELERY_TIMEZONE='Asia/Shanghai' 
CELERY_IMPORTS = ('celery_demo.tasks','celery_demo.tasks2')
# other config option

默认flower的配置文件名是flowerconfig.py,本例的内容如下：

broker_api = 'http://guest:guest[@localhost](https://my.oschina.net/u/570656):15672/api/'
logging = 'DEBUG'
basic_auth = ['admin:admin'] #加上授权保护

address = '127.0.0.1'
port = 5556

#开发任务 项目结构如下：

├── celery_demo
│   ├── celeryconfig.py 
│   ├── flowerconfig.py
│   ├── __init__.py 
│   ├── README.md
│   ├── tasks2.py 
│   ├── tasks.py 
├── __init__.py     
└── test
    ├── __init__.py 
    ├── tasks_test.py

为演示目的，tasks.py和tasks2.py的代码是相同的,tasks.py的代码如下：

from celery import Celery
from celery.exceptions import Reject,MaxRetriesExceededError
from celery.utils.log import task_logger

import celeryconfig
app = Celery()
import requests
from requests.exceptions import ConnectTimeout 

class ErrorURLStartException(Exception):
    def __init__(self):
        self.message = '必须是ＨＴＴＰ或ＨＴＴＰＳ开头' 

@app.task(bind=True, acks_late=True)
def task1(self, url):
    print url
    try:
        if url.startswith('http')==False :
            raise ErrorURLStartException()
        res = requests.get(url, timeout=5)
        task_logger.info('status_code:%s' % res.status_code)
    except ConnectTimeout as e:
        raise self.retry(exc=e, countdown=5, max_retries=5)
    except ErrorURLStartException as e:
        raise Reject(reason=e, requeue=False) #requeue=True可能会造成任务一致被循环的处理，永远不会结束
    else:
        task_logger.info('任务执行完毕') 

if __name__ == '__main__':
    app.config_from_object(celeryconfig)
    app.worker_main(['tasks', '-lDEBUG'])

tasks.task1任务的目的是当一个合法的HTTP URL过来的时候去GET内容，如果出现连接超时的异常就重试5次，如果出现非法的URL就直接将任务抛弃。

单元测试的代码如下：

import unittest
from unittest import TestCase 
from celery_demo.tasks import task1
from celery_demo.tasks2 import task1 as task2




class TasksTestCase(TestCase):
    def test_task1(self):
        print 'test_task1'
        url = 'http://www.baidu.com'
        task1.apply_async(args=[url])
     

    def test_task2(self):
        print 'test_task2'
        url = 'http://169.24.1.100'
        task1.apply_async(args=[url])
    

    def test_task3(self):
        print 'test_task3'
        url = 'adfhttp://169.24.1.100'
        task1.apply_async(args=[url])
    

    def test_task4(self):
        print 'test_task4'
        url = 'http://169.24.1.100'
        task2.apply_async(args=[url])

启动任务

进入到 celery_demo的上一级目录

george@george-pc:~/workspace/work_demo$ ll
总用量 24
drwxr-xr-x 2 george george 4096 9月  16 22:45 celery_demo
-rw-r--r-- 1 george george   56 9月  15 22:36 __init__.py
drwxr-xr-x 2 george george 4096 6月   4 11:29 rabbitmq_demo
-rw-r--r-- 1 george george  254 9月  15 21:40 tasks.py
-rw-r--r-- 1 george george  547 9月  16 00:12 tasks.pyc
drwxr-xr-x 2 george george 4096 9月  16 12:58 test

分别实行下面的三条指令:

启动flower的实例

celery flower --conf=celery_demo/flowerconfig.py
启动celery worker的实例

celery worker --config=celery_demo.celeryconfig -n worker1.%h -l DEBUG
运行单元测试

python -m unittest test.tasks_test