How to start a long-running process from a Django view?

Question

I need to run a process that might take hours to complete from a Django view. I don't need to know the state or communicate with it but I need that view to redirect away right after starting the process.

I've tried using subprocess.Popen, using it within a new threading.Thread, multiprocessing.Process. However, the parent process keeps hanging until child terminates. The only way that almost gets it done is using a fork. Obviously that isn't good as it leaves a zombie process behind until parent terminates.

That's what I'm trying to do when using fork:

if os.fork() == 0:
    subprocess.Popen(["/usr/bin/python", script_path, "-v"])
else:
    return HttpResponseRedirect(reverse('view_to_redirect'))

So, is there a way to run a completely independent process from a Django view with minimal casualties? Or am I doing something wrong?

Answer 1

I don't know if this will be suitable for your case, nevertheless here is what I do: I use a task queue (via a django model); when the view is called, it enters a new record in the tasks and redirects happily. Tasks in turn are executed by cron on a regular basis independently from django.

Edit: cron calls the relevant (and custom) django command to execute the task.

Answer 2

First of all - try to using cron for you task, as early say shanyu.

If it doesn't suit you - then try to use CeleryProject, for task Queue for Django. For working it uses RabbitMQ. And here is a little overview for simple using of basing futures

Answer 3

http://code.google.com/p/django-command-extensions/wiki/JobsScheduling

Is a nice library that that you can use to accomplish this task.

Answer 4

Take a look at the code in kronos.py to see one solution to this problem.

http://www.razorvine.net/download/kronos.py