some django's logs are missing when host in uwsgi with multiple process

Question

I'm using django+uwsgi for a web project. But I found some django logs would be missing after uwsgi is running for a while!

The situation is that: I config the uwsgi with 8 process. When I start the uwsgi, all the django logs would be written in single log file. But after a few hours, some logs are not written in file. I compared the django log file to uwsgi log file. I found only one uwsgi process's requests were written in django file. The other 7 process's django logs were missing. When I restart the uwsgi, it is the same result.

Does anyone know about this?

Thanks,

my django logging config:

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
    'simple': {
        'format': '%(levelname)s %(asctime)s %(message)s'
    },
    'detail': {
        'format': '%(levelname)s %(asctime)s [%(module)s.%(funcName)s line:%(lineno)d] %(message)s',
    },
},
'handlers': {
    'file': {
        'level': 'INFO',
        'formatter': 'simple',
        'class': 'logging.handlers.TimedRotatingFileHandler',
        'filename': LOG_FILE,
        'when': 'midnight',
        'backupCount': 366,
    },
    'err_file': {
        'level': 'WARN',
        'formatter': 'detail',
        'class': 'logging.handlers.TimedRotatingFileHandler',
        'filename': LOG_ERR_FILE,
        'when': 'midnight',
        'backupCount': 366,
    },
},
'loggers': {
    'django_request': {
        'handlers': ['file', 'err_file'],
        'level': 'DEBUG',
        'propagate': True,
    },
}
}

my uwsgi config:

<uwsgi>
<socket>0.0.0.0:8888</socket>
<chdir>src_dir</chdir>
<pythonpath>..</pythonpath>
<module>wsgi</module>
<workers>4</workers>
<processes>8</processes>
<master>true</master>
<pidfile>uwsgi.pid</pidfile>
<enable-threads>true</enable-threads>
<logdate>true</logdate>
<daemonize>/log/uwsgi/uwsgi.log</daemonize>
</uwsgi>

Answer 1

This is because it is not safe to write to the same file from several workers. You should write logs to different files from each worker, or try to use SysLogHandler

http://docs.python.org/library/logging.handlers.html

/etc/rsyslog.d/local-myapp.conf:

local0.*                         -/var/log/myapp/app.log

settings.py:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'user': {
            '()': 'core.log.UserFilter',
        }
    },
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(user)s %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(asctime)s %(module)s %(user)s %(message)s'
        },
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
            'filters': ['user'],
        },
        'syslog':{
            'level': 'INFO',
            'class': 'logging.handlers.SysLogHandler',
            'formatter': 'verbose',
            'filters': ['user'],
            'facility': 'local0',
            'address': '/dev/log',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    },
    'root': {
        'handlers': ['syslog'],
        'level': 'INFO',
    }
}

You also can try to write log messages into console and let uwsgi write the logs by himself.