celery - chaining groups and subtasks. -> out of order execution

Question

When I have something like the following

group1 = group(task1.si(), task1.si(), task1.si())
group2 = group(task2.si(), task2.si(), task2.si())

workflow = ch         


        

Answer 1

I have the same issue with celery, trying to have a workflow where the first step is "spawn a million tasks". Tried groups of groups, subtasks, eventually my step2 kicks off before step1 is over.

Long story short I might have found a solution with the use of chords and a dumb finisher:

@celery.task
def chordfinisher( *args, **kwargs ):
  return "OK"

Doing nothing much, but it enables me to do this:

tasks = []
for id in ids:
    tasks.append( mytask.si( id ) )
step1 = chord( group( tasks ), chordfinisher.si() )

step2 = ...

workflow = chain( step1, step2 )

Originally I wanted to have step1 in a subtask but for the same reason as suspected, the action of calling a group ends, the task is considered finished, and my workflow moves on...

If someone has something better, I'm interested!