AWS Elastic Beanstalk logging with python (django)

Question

How do you manage your application logs in AWS elastic beanstalk? I mean you write you application logs to which file? I'm using the following Logging configuration in my development environment but this doesn't work when I deploy in AWS.

Thanks in advance!

DEBUG_LOG_DIR = BASE_DIR + "/django_debug.log"
LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    # How to format the output
    'formatters': {
        'standard': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
    },
    # Log handlers (where to go)
    'handlers': {
        'null': {
            'level':'DEBUG',
            'class':'django.utils.log.NullHandler',
        },
        'log_file': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': DEBUG_LOG_DIR,
            'maxBytes': 50000,
            'backupCount': 2,
            'formatter': 'standard',
        },
        'console':{
            'level':'INFO',
            'class':'logging.StreamHandler',
            'formatter': 'standard'
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
        },
    },
    # Loggers (where does the log come from)
    'loggers': {
        'repackager': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers':['console'],
            'propagate': True,
            'level':'WARN',
        },
        'django.db.backends': {
            'handlers': ['console', 'log_file'],
            'level': 'WARN',
            'propagate': False,
        },
        '': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
        },
    }
}

Answer 1

I had a similar issue but on Elastic Beanstalk, so I created a config file (e.g. applogs.config) in .ebextensions folder of the app. This creates the app-logs folder if it is not there already and sets the file permissions and owner so that the app can write its logs there.

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

Finally, in your Django settings:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/app-logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Aditionally, if you want your log to be accessible from beanstalk logs using the web, add this to your file in .ebextensions

files:
  "/opt/elasticbeanstalk/tasks/taillogs.d/django.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/app-logs/django.log

Answer 2

Ok, I figured out a way to do it.

First I connected via ssh to ec2 machine, then I create a folder in /var/log called app_logs with root user:

mkdir /var/log/app_logs

After that I did the follow:

cd /var/log/
chmod g+s app_logs/
setfacl -d -m g::rw app_logs/
chown wsgi:wsgi app_logs/

That ensures that all the files created in this folder will have wsgi as owner and will be writable for the group that the file belongs. I had to do that because I noticed that the log file created by django app had root as owner and owner group but the application runs through wsgi user.

Finally I changed DEBUG_LOG_DIR to /var/log/app_logs/django_debug.log

Answer 3

There's a simple way that doesn't require any beanstalk configuration.

In your django settings under LOGGING set up a handler directed to the file '/opt/python/log/{log_file_name}'. The logs can then be accessed via the beanstalk environment menu under "Logs".

LOGGING = {
    ...,
    'handlers': {
        'logfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/opt/python/log/{log_file_name}',
        },
    },
    'loggers': {
        'debugger': {
            'level': 'DEBUG',
            'handlers': ['logfile'],
        'propagate': False,
    },
}

This location is stated in the documentation here:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-instancelocation

Answer 4

As a beginner in terms of linux permissions, it took me some time to get it to work. Summarising the above given answers the following finally worked for me:

logging.config

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_default_owner:
    command: setfacl -d -m g::rw /var/log/app-logs
  03_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

settings.py

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
    'file': {
        'level': 'DEBUG',
        'class': 'logging.FileHandler',
        'filename': '/var/log/app-logs/django.log',
    },
},
'loggers': {
    'django': {
        'handlers': ['file'],
        'level': 'DEBUG',
        'propagate': True,
    },
},
}

With this I can see the logs as a separate section using 'eb logs' or within the Beanstalk environment, section "logs".

Answer 5

By default in elasticbeanstalk, you can see the django error logs here.

/var/log/httpd/error_log