Python concurrent.futures using subprocess with a callback

Question

I am executing a FORTRAN exe from Python. The FORTRAN exe takes many minutes to complete; therefore, I need a callback to be fired when the exe finishes. The exe does not return anything back to Python, but in the callback function I will use Python to parse output text files from the FORTRAN.

For this I am using concurrent.futures and add_done_callback(), and it works. But this part of a web service and I need to have the Python method that calls subprocess.call() / Popen() to return once the FORTRAN exe is executed. Then when the FORTRAN is complete the callback function is called.

def fortran_callback(run_type, jid):

    return "Fortran finished executing"

def fortran_execute():

    from concurrent.futures import ThreadPoolExecutor as Pool

    args = "fortran.exe arg1 arg2"

    pool = Pool(max_workers=1)
    future = pool.submit(subprocess.call, args, shell=1)
    future.add_done_callback(fortran_callback(run_type, jid))
    pool.shutdown(wait=False)

    return "Fortran executed"

fortran_execute() is called when a form is submitted and I want to return "Fortran executed" without waiting for the FORTRAN to complete.

Currently the Python method returns without waiting for the FORTRAN to complete, but it also triggers the callback when it returns. The FORTRAN process continues to run, and when it eventually is completed it tries to call the callback function and an exception is thrown because the future is no longer present TypeError: 'NoneType' object is not callable.

What am I missing here to start an exe with subprocess, have the function return, and then have a callback method called only when the exe is finished executing?

Answer 1

Ok, now I know what you want and what your problem is.

def fortran_callback(future):
    print(future.run_type, future.jid)
    return "Fortran finished executing"

def fortran_execute():

    from concurrent.futures import ProcessPoolExecutor as Pool

    args = "sleep 2; echo complete"

    pool = Pool(max_workers=1)
    future = pool.submit(subprocess.call, args, shell=1)
    future.run_type = "run_type"
    future.jid = "jid"
    future.add_done_callback(fortran_callback)

    print("Fortran executed")


if __name__ == '__main__':
    import subprocess
    fortran_execute()

Run the above code gives output:

$ python3 test.py                                                                                                                                                      
Fortran executed
complete
run_type jid

1. Using ThreadPool is OK, but ProcessPool is better if fortran_callback is computationally expensive
2. Callback takes only one parameter which is the future object, so what you need to do is passing parameters via future's attributes.