Why no errors from multiprocessing is reported in Python and how to switch on reporting errors?

Question

I setup some simple code to test some problem handling with multiprocessing and I can not track the bug inside this code because not feedback from processes. How can I receive exception from subprocesses since now I am blind to it. How to debug this code.

# coding=utf-8
import multiprocessing
import multiprocessing.managers
import logging

def callback(result):
  print multiprocessing.current_process().name, 'callback', result

def worker(io_lock, value):
  # error
  raise RuntimeError()
  result = value + 1
  with io_lock:
    print multiprocessing.current_process().name, value, result
  return result

def main():
  manager = multiprocessing.Manager()
  io_lock = manager.Lock()
  pool = multiprocessing.Pool(multiprocessing.cpu_count())
  for i in range(10):
    print pool.apply_async(worker, args=(io_lock, i), callback = callback)
  pool.close()
  pool.join()

if __name__ == '__main__':
  logging.basicConfig(level = logging.DEBUG)
  main()

Answer 1

You can try, catch, and log exceptions that occur inside your worker processes. Something like this

def worker(io_lock, value):
    try:
        raise RuntimeError('URGH') # Or do your actual work here
    except:
        logging.exception('Exception occurred: ')
        raise  # Re-raise the exception so that the process exits 

The exception log handler will automatically include the stacktrace.