How to create Celery Windows Service?

Question

I\'m trying to create a Windows Service to launch Celery. I have come across an article that does it using Task Scheduler. However it seems to launch numerous celery instanc

Answer 1

The accepted answer does not apply for running celery with a Django application. But it inspired me to come up with a solution for running celery as a Windows service with Django. Note that the following is for Django projects only. It may work with other applications with some modifications.

The following discussion assumes Python >= 3.6 and RabbitMQ are already installed, and rabbitmq-server is running on localhost.

Create a file celery_service.py (or whatever you like) inside your Django project's top level folder, same level as manage.py, with the following content:

'''Usage : python celery_service.py install (start / stop / remove)
Run celery as a Windows service
'''
import win32service
import win32serviceutil
import win32api
import win32con
import win32event
import subprocess
import sys
import os
from pathlib import Path
import shlex
import logging
import time

# The directory for celery.log and celery_service.log
# Default: the directory of this script
INSTDIR = Path(__file__).parent
# The path of python Scripts
# Usually it is in path_to/venv/Scripts.
# If it is already in system PATH, then it can be set as ''
PYTHONSCRIPTPATH = INSTDIR / 'venvcelery/Scripts'
# The directory name of django project
# Note: it is the directory at the same level of manage.py
# not the parent directory
PROJECTDIR = 'proj'

logging.basicConfig(
    filename = INSTDIR / 'celery_service.log',
    level = logging.DEBUG, 
    format = '[%(asctime)-15s: %(levelname)-7.7s] %(message)s'
)

class CeleryService(win32serviceutil.ServiceFramework):

    _svc_name_ = "Celery"
    _svc_display_name_ = "Celery Distributed Task Queue Service"

    def __init__(self, args):
        win32serviceutil.ServiceFramework.__init__(self, args)
        self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)           

    def SvcStop(self):
        logging.info('Stopping {name} service ...'.format(name=self._svc_name_))        
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        win32event.SetEvent(self.hWaitStop)
        self.ReportServiceStatus(win32service.SERVICE_STOPPED)
        sys.exit()           

    def SvcDoRun(self):
        logging.info('Starting {name} service ...'.format(name=self._svc_name_))
        os.chdir(INSTDIR) # so that proj worker can be found
        logging.info('cwd: ' + os.getcwd())
        self.ReportServiceStatus(win32service.SERVICE_RUNNING)
        command = '"{celery_path}" -A {proj_dir} worker -f "{log_path}" -l info -P eventlet'.format(
            celery_path=PYTHONSCRIPTPATH / 'celery.exe',
            proj_dir=PROJECTDIR,
            log_path=INSTDIR / 'celery.log')
        logging.info('command: ' + command)
        args = shlex.split(command)
        proc = subprocess.Popen(args)
        logging.info('pid: {pid}'.format(pid=proc.pid))
        self.timeout = 3000
        while True:
            rc = win32event.WaitForSingleObject(self.hWaitStop, self.timeout)
            if rc == win32event.WAIT_OBJECT_0:
                # stop signal encountered
                # terminate process 'proc'
                PROCESS_TERMINATE = 1
                handle = win32api.OpenProcess(PROCESS_TERMINATE, False, proc.pid)
                win32api.TerminateProcess(handle, -1)
                win32api.CloseHandle(handle)                
                break

if __name__ == '__main__':
   win32serviceutil.HandleCommandLine(CeleryService)

Before the script can be run, you need to

1. Optionally create a python virtual environment e.g. 'venvcelery'.

2. Install the following requirements:

   django>=2.0.0 sqlalchemy>=1.0.14 celery>=4.3.0,<5.0 pywin32>=227 eventlet>=0.25

3. Fix pywin32 pywintypes36.dll location. ref

4. Correctly set PYTHONSCRIPTPATH and PROJECTDIR in celery_service.py

PYTHONSCRIPTPATH is usually the "Scripts" folder under your python's installation path or current virtual environment

PROJECTDIR is the directory name of the Django project.

It is the directory at the same level of manage.py, not the parent directory.

Now you can install / start / stop / remove the service with:

python celery_service.py install
python celery_service.py start
python celery_service.py stop
python celery_service.py remove

I created a demo Django project with celery running as a Windows service:

https://github.com/azalea/django_celery_windows_service

In case you are interested in a running example.

---

Note: this is an updated version assuming Python >= 3.6, Django 2.2 and Celery 4.

An older version with Python 2.7, Django 1.6 and Celery 3 can be viewed in edit history.