How do I run a long-running job in the background in Python

Question

I have a web-service that runs long-running jobs (in the order of several hours). I am developing this using Flask, Gunicorn, and nginx.

What I am thinking of doing

Answer 1

Your approach is fine and will totally work, but why reinvent the background worker for python web applications when a widely accepted solution exists, namely celery.

I'd need to see a lot tests before I trusted any home rolled code for such an important task.

Plus celery gives you features like task persistence and the ability to distribute workers across multiple machines.