Retry Celery tasks with exponential back off

Question

For a task like this:

from celery.decorators import task

@task()
def add(x, y):
    if not x or not y:
        raise Exception(\"test error\")
    return se         


        

Answer 1

The task.request.retries attribute contains the number of tries so far, so you can use this to implement exponential back-off:

from celery.task import task

@task(bind=True, max_retries=3)
def update_status(self, auth, status):
    try:
        Twitter(auth).update_status(status)
    except Twitter.WhaleFail as exc:
        self.retry(exc=exc, countdown=2 ** self.request.retries)

To prevent a Thundering Herd Problem, you may consider adding a random jitter to your exponential backoff:

import random
self.retry(exc=exc, countdown=int(random.uniform(2, 4) ** self.request.retries))