Monitoring the asyncio event loop

Question

I am writing an application using python3 and am trying out asyncio for the first time. One issue I have encountered is that some of my coroutines block the event loop for longer than I like. I am trying to find something along the lines of top for the event loop that will show how much wall/cpu time is being spent running each of my coroutines. If there isn't anything already existing does anyone know of a way to add hooks to the event loop so that I can take measurements?

I have tried using cProfile which gives some helpful output, but I am more interested in time spent blocking the event loop, rather than total execution time.

Answer 1

Event loop can already track if coroutines take much CPU time to execute. To see it you should enable debug mode with set_debug method:

import asyncio
import time


async def main():
    time.sleep(1)  # Block event loop


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.set_debug(True)  # Enable debug
    loop.run_until_complete(main())

In output you'll see:

Executing <Task finished coro=<main() [...]> took 1.016 seconds

By default it shows warnings for coroutines that blocks for more than 0.1 sec. It's not documented, but based on asyncio source code, looks like you can change slow_callback_duration attribute to modify this value.

Answer 2

You can use call_later. Periodically run callback that will log/notify the difference of loop's time and period interval time.

class EventLoopDelayMonitor:

    def __init__(self, loop=None, start=True, interval=1, logger=None):
        self._interval = interval
        self._log = logger or logging.getLogger(__name__)
        self._loop = loop or asyncio.get_event_loop()
        if start:
            self.start()

    def run(self):
        self._loop.call_later(self._interval, self._handler, self._loop.time())

    def _handler(self, start_time):
        latency = (self._loop.time() - start_time) - self._interval
        self._log.error('EventLoop delay %.4f', latency)
        if not self.is_stopped():
            self.run()

    def is_stopped(self):
        return self._stopped

    def start(self):
        self._stopped = False
        self.run()

    def stop(self):
        self._stopped = True

example

import time

async def main():
    EventLoopDelayMonitor(interval=1)
    await asyncio.sleep(1)
    time.sleep(2)
    await asyncio.sleep(1)
    await asyncio.sleep(1)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

output

EventLoop delay 0.0013
EventLoop delay 1.0026
EventLoop delay 0.0014
EventLoop delay 0.0015

Answer 3

To expand a bit on one of the answers, if you want to monitor your loop and detect hangs, here's a snippet to do just that. It launches a separate thread that checks whether the loop's tasks yielded execution recently enough.

def monitor_loop(loop, delay_handler):
loop = loop
last_call = loop.time()

INTERVAL = .5  # How often to poll the loop and check the current delay.
def run_last_call_updater():
    loop.call_later(INTERVAL, last_call_updater)
def last_call_updater():
    nonlocal last_call
    last_call = loop.time()
    run_last_call_updater()
run_last_call_updater()

def last_call_checker():
    threading.Timer(INTERVAL / 2, last_call_checker).start()
    if loop.time() - last_call > INTERVAL:
        delay_handler(loop.time() - last_call)
threading.Thread(target=last_call_checker).start()