How do I run a long-running job in the background in Python

Question

I have a web-service that runs long-running jobs (in the order of several hours). I am developing this using Flask, Gunicorn, and nginx.

What I am thinking of doing

Answer 1

Well, Although your approach is not incorrect, basicly it may lead your program run out of available threads. As Ali mentioned, a general approach is to use Job Queues like RQ or Celery. However you don't need to extract functions to use those libraries. For Flask, I recommend you to use Flask-RQ. It's simple to start:

RQ

pip install flask-rq

Just remember to install Redis before using it in your Flask app.

And simply use @Job Decorator in your Flask functions:

from flask.ext.rq import job


@job
def process(i):
    #  Long stuff to process


process.delay(3)

And finally you need rqworker to start the worker:

rqworker

You can see RQ docs for more info. RQ designed for simple long running processes.

Celery

Celery is more complicated, has huge list of features and is not recommended if you are new to job queues and distributed processing methods.

Greenlets

Greenlets have switches. Let you to switch between long running processes. You can use greenlets for running processes. The benefit is you don't need Redis and other worker, instead you have to re-design your functions to be compatible:

from greenlet import greenlet

def test1():
    print 12
    gr2.switch()
    print 34

def test2():
    print 56
    gr1.switch()
    print 78

gr1 = greenlet(test1)
gr2 = greenlet(test2)
gr1.switch()