Django 2.0 + Celery 3.1.25 定时（周期）任务

1.目的

　　最近根据项目需要，希望在django框架下实现一些定时任务功能

　　此为demo示例，简单的做每5s 实现 'hello world' 的打印

　　踩过一些坑，最终还是顺利完成了配置。

2.基建

　　使用的包：[django==2.0, django-celery==3.2.2, celery==3.1.25]

　　创建Django项目以及相关app

　　在新建的app目录下添加tasks文件夹

　　在settings.py所在同级目录新建 celery.py

C:.
│  celery
│  db.sqlite3
│  manage.py
│  requirements.txt
│
├─app_name
│  │  admin.py
│  │  apps.py
│  │  models.py
│  │  tests.py
│  │  views.py
│  │  __init__.py
│  │
│  ├─migrations
│  │  │  __init__.py
│  │
│  ├─tasks
│  │  │  timer_job.py
│  │  │  __init__.py
│
├─project_name
│  │  celery.py
│  │  settings.py
│  │  urls.py
│  │  wsgi.py
│  │  __init__.py

　　创建django-celery所需要的表，在控制台执行

　　python manage.py migrate djcelery

3.配置

　　关于任务注册，我选择在celery.py里进行

　　配置settings.py

　　PS: broker使用 rabbitmq 3.8.1 安装完成后，确定可以登录管理界面 15672即可

　　　　backend 和调度器都与django orm关联

　　配置celery.py 参考：http://docs.celeryproject.org/en/3.1/django/first-steps-with-django.html

 1 from __future__ import absolute_import
 2 
 3 import os
 4 
 5 from celery import Celery
 6 
 7 from django.conf import settings
 8 
 9 from datetime import timedelta
10 
11 #from celery.schedules import crontab
12 
13 os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project_name.settings')
14 
15 app = Celery('project_name')
16 
17 app.config_from_object('django.conf:settings')
18 
19 app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
20 
21 #此处使用 beat.schedule 无法完成注册
22 #任务注册也可在 settings里完成
23 app.conf.update(CELERYBEAT_SCHEDULE={
24     'hello': {
25         'task': 'app_name.tasks.timer_job.hello',
26         'schedule': timedelta(second=10),
27         'args': (),
28     },
29 })
30 
31 app.conf.timezone = 'UTC'

　　在project_name下的 __init__.py 导入celery实例

from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app

4.任务

　　在app tasks文件夹下添加将要执行的任务 timer_job.py

from project_name import celery_appdef say_hello():
    return 'hello world'

@celery_app.task
def hello():
    try:
        a = say_hello()
        print(a)
    except Exception:
        print('调用方法错误')

　　在tasks文件夹下添加 __init__.py

　　(根据上面配置 worker查找注册任务本应该查找每个app下 tasks.py 但此处使用tasks文件夹以应对添加多任务的情况，需要把tasks转为py包)

from .timer_job import hello

5.启动

　　执行路径改为 celery.py的上级目录

　　在控制台中执行 python manage.py celery beat

celery beat v3.1.25 (Cipater) is starting.
__    -    ... __   -        _
Configuration ->
    . broker -> amqp://guest:**@localhost:5672//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> djcelery.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> now (0s)
[2019-11-15 13:54:01,998: INFO/MainProcess] beat: Starting...
[2019-11-15 13:54:01,998: INFO/MainProcess] Writing entries (0)...
[2019-11-15 13:54:02,249: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2019-11-15 13:54:02,250: INFO/MainProcess] Writing entries (0)...
[2019-11-15 13:54:02,283: INFO/MainProcess] Scheduler: Sending due task hello (app_name.tasks.timer_job.hello)
[2019-11-15 13:54:02,309: INFO/MainProcess] Writing entries (1)...
[2019-11-15 13:54:12,284: INFO/MainProcess] Scheduler: Sending due task hello (app_name.tasks.timer_job.hello)
[2019-11-15 13:54:22,284: INFO/MainProcess] Scheduler: Sending due task hello (app_name.tasks.timer_job.hello)
[2019-11-15 13:54:32,285: INFO/MainProcess] Scheduler: Sending due task hello (app_name.tasks.timer_job.hello)
[2019-11-15 13:54:42,285: INFO/MainProcess] Scheduler: Sending due task hello (app_name.tasks.timer_job.hello)
[2019-11-15 13:54:52,286: INFO/MainProcess] Scheduler: Sending due task hello (app_name.tasks.timer_job.hello)

　　可以看到提示消息每10s向队列发送任务信息

　　执行路径改为 celery.py的上级目录

　　在新控制台中执行 python manage.py celery -A project_name worker -l info

  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

 -------------- celery@PUZHA-PS6NN v3.1.25 (Cipater)
---- **** -----
--- * ***  * -- Windows-10-10.0.17134-SP0
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         project_name:0xf4f250
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . app_name.tasks.timer_job.hello

[2019-11-15 13:55:10,912: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-11-15 13:55:10,974: INFO/MainProcess] mingle: searching for neighbors
[2019-11-15 13:55:12,080: INFO/MainProcess] mingle: all alone
[2019-11-15 13:55:12,280: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[05202beb-d26e-433e-9894-83e9eb9a5c4c]
[2019-11-15 13:55:12,281: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[3e3587fa-5352-4c55-bb89-1e7f4450da29]
[2019-11-15 13:55:12,282: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[6e8072c1-765d-445e-a880-4d81a49980c8]
[2019-11-15 13:55:12,283: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[02a8c7ac-2917-4919-9c8b-a51039ce01ed]
[2019-11-15 13:55:12,284: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[37e2d2e4-f7f6-4e40-a5b4-f03c5a439ff4]
[2019-11-15 13:55:12,285: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[fe6a126e-dcd5-4589-816b-04e2ee011050]
[2019-11-15 13:55:12,291: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[5d80097d-5f9c-4141-af1a-2c5da7fe61f9]
[2019-11-15 13:55:12,309: INFO/MainProcess] Received task: app_name.tasks.timer_job.hello[ffe49ee4-895a-4481-a0ac-edbf8b3c528b]
[2019-11-15 13:55:15,361: INFO/Worker-1] child process 14108 calling self.run()
[2019-11-15 13:55:15,407: INFO/Worker-1] child process 12952 calling self.run()
[2019-11-15 13:55:15,447: INFO/Worker-1] child process 22160 calling self.run()
[2019-11-15 13:55:15,560: INFO/Worker-1] child process 948 calling self.run()
[2019-11-15 13:55:16,123: INFO/Worker-1] child process 18836 calling self.run()
[2019-11-15 13:55:16,141: INFO/Worker-1] child process 18416 calling self.run()
[2019-11-15 13:55:16,476: INFO/Worker-1] child process 18776 calling self.run()
[2019-11-15 13:55:16,570: INFO/Worker-1] child process 21080 calling self.run()
[2019-11-15 13:55:19,363: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,365: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,364: INFO/MainProcess] Task app_name.tasks.timer_job.hello[05202beb-d26e-433e-9894-83e9eb9a5c4c] succeeded in 0s: None
[2019-11-15 13:55:19,368: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,379: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,380: INFO/MainProcess] Task app_name.tasks.timer_job.hello[3e3587fa-5352-4c55-bb89-1e7f4450da29] succeeded in 0.014999999999417923s: None
[2019-11-15 13:55:19,380: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,381: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,381: INFO/MainProcess] Task app_name.tasks.timer_job.hello[6e8072c1-765d-445e-a880-4d81a49980c8] succeeded in 0.014999999999417923s: None
[2019-11-15 13:55:19,382: WARNING/Worker-1] hello world
[2019-11-15 13:55:19,382: WARNING/Worker-1] hello world