How to cancel long-running subprocesses running using `concurrent.futures.ProcessPoolExecutor`?

Question

You can see the full here.

A simplified version of my code follows:

executor = ProcessPoolExecutor(10)
try:
    coro = bot.loop.run_in_executor(executor, processUserInput, userInput)
    result = await asyncio.wait_for(coro, timeout=10.0, loop=bot.loop)
except asyncio.TimeoutError:
    result="Operation took longer than 10 seconds. Aborted."

Unfortunately, when an operation times out, that process is still running, even though the future has been cancelled. How do I cancel that process/task so that it actually stops running?

Answer 1

ProcessPoolExecutor uses the multiprocessing module. Instead of canceling the event, which does not .terminate() the subprocess, It is recommended to use a multiprocessing.Event to allow your subprocess to exit properly:

import asyncio
import multiprocessing
import time
from concurrent.futures.process import ProcessPoolExecutor


def f(done):
    print("hi")

    while not done.is_set():
        time.sleep(1)
        print(".")

    print("bye")

    return 12345


async def main():
    done = manager.Event()
    fut = loop.run_in_executor(None, f, done)
    print("waiting...")
    try:
        result = await asyncio.wait_for(asyncio.shield(fut), timeout=3)
    except asyncio.TimeoutError:
        print("timeout, exiting")
        done.set()
        result = await fut
    print("got", result)

loop = asyncio.get_event_loop()
loop.set_default_executor(ProcessPoolExecutor())
manager = multiprocessing.Manager()

loop.run_until_complete(main())