Python Asyncio run_forever() and Tasks

Question

I adapted this code for using Google Cloud PubSub in Async Python: https://github.com/cloudfind/google-pubsub-asyncio

import asyncio
import datetime
import f         


        

Answer 1

PubSub is calling the create_proc_message_task callback from a different thread. Since create_task is not thread-safe, it must only be called from the thread that runs the event loop (typically the main thread). To correct the issue, replace loop.create_task(proc_message(message)) with asyncio.run_coroutine_threadsafe(proc_message(message), loop) and message_producer will no longer be needed.

As for why message_producer appeared to fix the code, consider that run_coroutine_threadsafe does two additional things compared to create_task:

• It operates in a thread-safe fashion, so the event loop data structures are not corrupted when this is done concurrently.
• It ensures that the event loop wakes up at the soonest possible opportunity, so that it can process the new task.

In your case create_task added the task to the loop's runnable queue (without any locking), but failed to ensure the wakeup, because that is not needed when running in the event loop thread. The message_producer then served to force the loop to wake up in regular intervals, which is when it also checks and executes the runnable tasks.