No logs appear on Cloudwatch log group for elastic beanstalk environment

Question

I have an elastic beanstalk environment, which is running a docker container that has a node js API. On the AWS Console, if I select my environment, then go to Configuration/Software I have the following:

• Log groups: /aws/elasticbeanstalk/my-environment
• Log streaming: Enabled
• Retention: 3 days
• Lifecycle: Keep after termination.

However, if I click on that log group on the Cloudwatch console, I have a Last Event Time of some weeks ago (which I believe corresponds to when the environment was created) and have no content on the logs.

Since this is a dockerized application, Logs for the server itself should be at /aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.log. If I instead get the Logs directly from the instances by going once again to my EB environment, clicking "Logs" and then "Request last 100 Lines" the logging is happening correctly. I just can't see a thing when using CloudWatch.

Any help is gladly appreciated

Answer 1

I was able to get around this problem. So CloudWatch makes a hash based on the first line of your log file and the log stream key, and the problem is that my first line on the stdouterr.log file was actually an empty line!

After couple of days playing around and getting help from the good AWS support team, I first connected via SSH to my EC2 instance associated to the EB environment and you need to add the following line to the /etc/awslogs/config/beanstalklogs.conf file, right after the "file=/var/log/eb-docker/containers/eb-current-app/stdouterr.log" line:

file_fingerprint_lines=1-20

With these, you tell the AWS service that it should calculate the hash using lines 1 through 20 on the log file. You could change 20 for larger or smaller numbers depending on your logging content; however I don't know if there is an upper limit for the value.

After doing so, you need to restart the AWS Logs Service on the instance.

For this you would execute:

• sudo service awslogs stop
• sudo service awslogs start

or simpler:

sudo service awslogs restart

After these steps I started using my environment and the logging was now being properly streamed to the CloudWatch console! However this would not work if a new deployment is made, if the EC2 instance gets replaced or the auto scalable group spawns another.

To have a fix for this, it is possible to add log config via the .ebextensions directory, at the root of your application before deploying.

I added a file called logs.config to the newly created .ebextensions directory and placed the following content:

files:
  "/etc/awslogs/config/beanstalklogs.conf":
    mode: "000644"
    user: root
    group: root
    content: |
      [/var/log/eb-docker/containers/eb-current-app/stdouterr.log]
      log_group_name=/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log
      log_stream_name={instance_id}
      file=/var/log/eb-docker/containers/eb-current-app/*stdouterr.log
      file_fingerprint_lines=1-20

commands:
  01_remove_eb_stream_config:
    command: 'rm /etc/awslogs/config/beanstalklogs.conf.bak'
  02_restart_log_agent:
    command: 'service awslogs restart'

Changing of course EB-ENV-NAME by my environment name on EB.

Hope it can help someone else!