Celery Storing unrecoverable task failures for later resubmission

Question

I'm using the djkombu transport for my local development, but I will probably be using amqp (rabbit) in production.

I'd like to be able to iterate over failures of a particular type and resubmit. This would be in the case of something failing on a server or some edge case bug triggered by some new variation in data.

So I could be resubmitting jobs up to 12 hours later after some bug is fixed or a third party site is back up.

My question is: Is there a way to access old failed jobs via the result backend and simply resubmit them with the same params etc?

Answer 1

You can probably access old jobs using:

CELERY_RESULT_BACKEND = "database"

and in your code:

from djcelery.models import TaskMeta
task = TaskMeta.objects.filter(task_id='af3185c9-4174-4bca-0101-860ce6621234')[0]

but I'm not sure you can find the arguments that the task is being started with ... Maybe something with TaskState...

I've never used it this way. But you might want to consider the task.retry feature? An example from celery docs:

@task()
def task(*args):
    try:
        some_work()
    except SomeException, exc:
        # Retry in 24 hours.
        raise task.retry(*args, countdown=60 * 60 * 24, exc=exc)

Answer 2

From IRC

<asksol> dpn`: task args and kwargs are not stored with the result

<asksol> dpn`: but you can create your own model and store it there (for example using the task_sent signal)

<asksol> we don't store anything when the task is sent, only send a message. but it's very easy to do yourself

This was what I was expecting, but hoped to avoid.

At least I have an answer now :)