Multiple Docker containers and Celery

Question

We have the following structure of the project right now:

1. Web-server that processes incoming requests from the clients.
2. Analytics module that provides

Answer 1

First, let clarify the difference between celery library (which you get with pip install or in your setup.py) and celery worker - which is the actual process that dequeue tasks from the broker and handle them. Of course you might wanna have multiple workers/processes (for separating different task to a different worker - for example).

Lets say you have two tasks: calculate_recommendations_task and periodic_update_task and you want to run them on a separate worker i.e recommendation_worker and periodic_worker. Another process will be celery beat which just enqueue the periodic_update_task into the broker each x hours.

In addition, let's say you have simple web server implemented with bottle.

I'll assume you want to use celery broker & backend with docker too and I'll pick the recommended usage of celery - RabbitMQ as broker and Redis as backend.

So now we have 6 containers, I'll write them in a docker-compose.yml:

version: '2'
services:
  rabbit:
    image: rabbitmq:3-management
    ports:
      - "15672:15672"
      - "5672:5672"
    environment:
      - RABBITMQ_DEFAULT_VHOST=vhost
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest
  redis:
    image: library/redis
    command: redis-server /usr/local/etc/redis/redis.conf
    expose:
      - "6379"
    ports:
      - "6379:6379"
  recommendation_worker:
    image: recommendation_image
    command: celery worker -A recommendation.celeryapp:app -l info -Q recommendation_worker -c 1 -n recommendation_worker@%h -Ofair
  periodic_worker:
    image: recommendation_image
    command: celery worker -A recommendation.celeryapp:app -l info -Q periodic_worker -c 1 -n periodic_worker@%h -Ofair
  beat:
    image: recommendation_image
    command: <not sure>
  web:
    image: web_image
    command: python web_server.py

both dockerfiles, which builds the recommendation_image and the web_image should install celery library. Only the recommendation_image should have the tasks code because the workers are going to handle those tasks:

RecommendationDockerfile:

FROM python:2.7-wheezy
RUN pip install celery
COPY tasks_src_code..

WebDockerfile:

FROM python:2.7-wheezy
RUN pip install celery
RUN pip install bottle 
COPY web_src_code..

The other images (rabbitmq:3-management & library/redis are available from docker hub and they will be pulled automatically when you run docker-compose up).

Now here is the thing: In you web server you can trigger celery tasks by their string name and pull the result by task-ids (without sharing the code) web_server.py:

import bottle
from celery import Celery
rabbit_path = 'amqp://guest:guest@rabbit:5672/vhost'
celeryapp = Celery('recommendation', broker=rabbit_path)
celeryapp.config_from_object('config.celeryconfig')

@app.route('/trigger_task', method='POST')
def trigger_task():
    r = celeryapp.send_task('calculate_recommendations_task', args=(1, 2, 3))
    return r.id

@app.route('/trigger_task_res', method='GET')
def trigger_task_res():
    task_id = request.query['task_id']
    result = celery.result.AsyncResult(task_id, app=celeryapp)
    if result.ready():
        return result.get()
    return result.state

last file config.celeryconfig.py:

CELERY_ROUTES = {
    'calculate_recommendations_task': {
        'exchange': 'recommendation_worker',
        'exchange_type': 'direct',
        'routing_key': 'recommendation_worker'
    }
}
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']