Rotate logfiles each time the application is started (Python)

Question

I\'m using the logging module in Python and I would like it to create a new logfile each time my application is started. The older logfiles shoud be rotated (eg: logfile.txt

Answer 1

Log Rotation and RoatatingFileHandler are usually designed and desirable when the application is running for a very long time (days) and you want the log to keep rotation. Under cases where I have to rotate the log upon restart of the application, I had to do that outside of the Logfile handler, which was easier. It was like, before the log writer call for the first time, I would see if the log file already existed, and if yes, rename it and create a new log file. The renaming should be differentiated from the handler's renaming mechanism.

Answer 2

Simplest way is just to have a date tag in log file name, so when you start app each time you will get a new log file.

e.g.

dateTag = datetime.datetime.now().strftime("%Y-%b-%d_%H-%M-%S")
logging.basicConfig(filename="myapp_%s.log" % dateTag, level=logging.DEBUG)

so each time you will have log like myapp_2011-Jan-11_12-27-29.log

Another benefit is that you can mix this with RotatingFileHandler to have separate log for each app invocation, where each log itself is further divided into multiple fixed size logs.

Answer 3

I might be enough to use RotatingFileHandler without maxBytes, then call doRollover() on application start.

Yup, seems to work fine. The code below will create a new log file on each application run, with added timestamps for log start and close times. Running it will print the list of available log files. You can inspect them to check correct behavior. Adapted from the Python docs example:

import os
import glob
import logging
import logging.handlers
import time

LOG_FILENAME = 'logging_rotatingfile_example.out'

# Set up a specific logger with our desired output level
my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)

# Check if log exists and should therefore be rolled
needRoll = os.path.isfile(LOG_FILENAME)

# Add the log message handler to the logger
handler = logging.handlers.RotatingFileHandler(LOG_FILENAME, backupCount=50)

my_logger.addHandler(handler)

# This is a stale log, so roll it
if needRoll:    
    # Add timestamp
    my_logger.debug('\n---------\nLog closed on %s.\n---------\n' % time.asctime())

    # Roll over on application start
    my_logger.handlers[0].doRollover()

# Add timestamp
my_logger.debug('\n---------\nLog started on %s.\n---------\n' % time.asctime())

# Log some messages
for i in xrange(20):
    my_logger.debug('i = %d' % i)

# See what files are created
logfiles = glob.glob('%s*' % LOG_FILENAME)

print '\n'.join(logfiles)

Answer 4

I had a similar requirement to be able to force a log rotation at startup based on a command-line option, but for the logfiles to otherwise rotate on their regular schedule. This was my solution:

import logging

from logging.handlers import BaseRotatingHandler
from typing import Union

def rotate_logs(loggers: Union[str,list]=None, delimiter: str=','):
    """Rotate logs.

    Args:
        loggers: List of logger names as list object or as string,
            separated by `delimiter`.

        delimiter: Separator for logger names, if `loggers` is :obj:`str`.
            Defaults to ``,`` (comma).

    """

    # Convert loggers to list.
    if isinstance(loggers, str):
        loggers = [t.strip() for t in loggers.split(delimiter)]

    handlers = []
    root = logging.getLogger()

    # Include root logger in dict.    
    ld = {'': root, **root.manager.loggerDict}

    for k, v in ld.items():
        if loggers is not None and k not in loggers:
            continue

        try:
            for h in v.handlers:
                if (isinstance(h, BaseRotatingHandler) and
                    h not in handlers):

                    handlers.append(h)

        except AttributeError:
            pass

    for h in handlers:
        h.doRollover()



if __name__ == '__main__':
    pass

Notes:

• This has been validated to work if maxBytes > 0 on a RotatingFileHandler.

• This method hasn't been tested with a TimedRotatingFileHandler, but should work.

• This method eliminates the need to maintain a reference to the RotatingFileHandler to be rotated; as a result, it can easily be used when configuring logging using logging.config.