Same task executed multiple times

问题

I have ETA tasks that get sent to a Redis broker for Celery. It is a single celery and redis instance, both int he same machine.

The problem is, tasks are getting executed multiple times. I've seen tasks executed 4 to 11 times.

I set up the visibility timeout to be 12 hours, given that my ETA's are between 4-11 hours (determined at runtime):

BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 12 * 60 * 60}

Even with that, tasks still get executed multiple times.

Initially, the task in question was not idempotent, so I tried adding in a DB check to make them idempotent.

it looks something like this:

@app.task
def foo(side_effect_action):
    if side_effect_action.executed:
        return ALREADY_EXECUTED
    else:
        do_side_effect()
        side_effect_action.executed = True
        side_effect_action.save() #hits the db
        return JUST_EXECUTED

Turns out that the celery worker gets to the task before foo is able to call side_effect_action.save() and save the state, so in all cases when it's looking for side_effect_action.executed it is still False, and thus gets executed multiple times.

Any ideas how can I solve this issue?

回答1:

I switched my celery broker to RabbitMQ to avoid this issue. It is unfortunate since I now have one more component in my webapp (I still need redis for something else), but it did solve the multiple execution for ETA tasks bug.

来源：https://stackoverflow.com/questions/21568354/same-task-executed-multiple-times

标签

python

Redis

celery