Constantly monitor a program/process using Python

Question

I am trying to constantly monitor a process which is basically a Python program. If the program stops, then I have to start the program again. I am using another Python program to do so.

For example, say I have to constantly run a process called run_constantly.py. I initially run this program manually, which writes its process ID to the file "PID" (in the location out/PROCESSID/PID).

Now I run another program which has the following code to monitor the program run_constantly.py from a Linux environment:

def Monitor_Periodic_Process():

    TIMER_RUNIN = 1800
    foo = imp.load_source("Run_Module","run_constantly.py")
    PROGRAM_TO_MONITOR = ['run_constantly.py','out/PROCESSID/PID']
    while(1):
        # call the function checkPID to see if the program is running or not
        res = checkPID(PROGRAM_TO_MONITOR)
        # if res is 0 then program is not running so schedule it
        if (res == 0):
            date_time = datetime.now()
            scheduler.add_cron_job(foo.Run_Module, year=date_time.year, day=date_time.day, month=date_time.month, hour=date_time.hour, minute=date_time.minute+2)
            scheduler.start()
            scheduler.get_jobs()
            time.sleep(TIMER_NOT_RUNIN)
            continue
        else:
            #the process is running sleep and then monitor again
            time.sleep(TIMER_RUNIN)
            continue

I have not included the checkPID() function here. checkPID() basically checks if the process ID still exists (i.e. if the program is still running) and if it does not exist, it returns 0. In the above program, I check if res == 0, and if so, then I use Python's scheduler to schedule the program. However, the major problem that I am currently facing is that the process ID of this program and the run_constantly.py program turns to be same once I schedule the run_constantly.py using the scheduler.add_cron_job() function. So if the program run_constantly.py crashes, the following program still thinks that the run_constantly.py is running (since both process IDs are same), and therefore continues to go into the else loop to sleep and monitor again.

Can someone tell me how to solve this issue? Is there a simple way to constantly monitor a program and reschedule it when it has crashed?

Answer 1

There are many programs that can do this.

On Ubuntu there is upstart (installed by default)

Lots of people like http://supervisord.org/

monit as mentioned by @nathan

If you are looking for a python alternative there is a library that has just been released called circus which looks interesting.

And pretty much every linux distro probably has one of these built in.

The choice is really just down to which one you like better, but you would be far better off using one of these than writing it yourself.

Hope that helps

Answer 2

If you are willing to control the monitored program directly from python instead of using cron, have a look at the subprocess module :

The subprocess module allows you to spawn new processes,
connect to their input/output/error pipes, and obtain their return codes.

Check examples like track process status with python on SO for examples and references.

Answer 3

You could just use monit http://mmonit.com/monit/

It monitors processes and restarts them (and other things.)