Aiohttp logging: how to distinguish log messages of different requests?

Question

Imagine I have this web application based on Aiohttp:

from aiohttp import web
import asyncio
import logging

logger = logging.getLogger(__name__)

async def          


        

Answer 1

In "classic" non-async web apps it's simple - one process (or thread) processes only one request at a time, so you just log process/thread id (logging format: %(process)d %(thread)d).

In async (asyncio) programs there are usually multiple different things running in an event loop in a single thread (in web app: different requests being processed), so logging process/thread id is not enough. You need to somehow identify not an operating system thread, but a "thread" of related asyncio tasks - that's what ContextVar is for.

Step 1: create contextvar

request_id = ContextVar('request_id')

Step 2: set this contextvar value for each request

@web.middleware
async def add_request_id_middleware(request, handler):
    '''
    Aiohttp middleware that sets request_id contextvar and request['request_id']
    to some random value identifying the given request.
    '''
    req_id = secrets.token_urlsafe(5).replace('_', 'x').replace('-', 'X')
    request['request_id'] = req_id
    token = request_id.set(req_id)
    try:
            return await handler(request)
    finally:
        request_id.reset(token)

app = web.Application(middlewares=[add_request_id_middleware])

Step 3: insert this contextvar value in every log message automatically

def setup_log_record_factory():
    '''
    Wrap logging request factory so that [{request_id}] is prepended to each message
    '''
    old_factory = logging.getLogRecordFactory()

    def new_factory(*args, **kwargs):
        record = old_factory(*args, **kwargs)
        req_id = request_id.get(None)
        if req_id:
            record.msg = f'[{req_id}] {record.msg}'
        return record

    logging.setLogRecordFactory(new_factory)

setup_log_record_factory()

Step 4: since aiohttp request access log message is logged outside the scope where we set the context var, we need to define our own AccessLogger that fixes this:

from aiohttp.web_log import AccessLogger

class CustomAccessLogger (AccessLogger):

    def log(self, request, response, time):
        token = request_id.set(request['request_id'])
        try:
            super().log(request, response, time)
        finally:
            request_id.reset(token)

web.run_app(app, access_log_class=CustomAccessLogger)

Done