How to know the reason why a docker container exits?

Question

I have a Docker container running in a host of 1G RAM (there are also other containers running in the same host). The application in this Docker container will decode some images, which may consume memory a lot.

From time to time, this container will exit. I doubt it is due to out of memory but not very sure. I need a method to find the root cause. So is there any way to know what happened for this container's death?

Answer 1

Others have mentioned docker logs $container_id to view the output of the application. This would always be my first thing to check.

Next, you can run a docker inspect $container_id to view details on the state, e.g.:

    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 2,
        "Error": "",
        "StartedAt": "2016-06-28T21:26:53.477229071Z",
        "FinishedAt": "2016-06-28T21:26:53.478066987Z"
    },

The important line there is "OOMKilled" which will be true if you exceed the container memory limits and Docker kills your app. You may also want to lookup the exit code to see if it identifies a cause for the exit by your app.

Note, this only indicates if docker itself kills your process, and requires that you have set a memory limit on your container. Outside of docker, the Linux kernel can lol your process if the host itself runs out of memory. Linux often writes to a log in /var/log when this happens. With Docker Desktop on Windows and Mac, you can adjust the memory allocated to the embedded Linux VM in the docker settings.

Answer 2

You can find out whether the process inside the container was OOMkilled by reading the logs. OOMkills are initiated by the kernel so every time it happens there's a bunch of lines in /var/log/kern.log, for example:

python invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
oom_kill_process+0x22e/0x450
Memory cgroup out of memory: Kill process 31204 (python) score 1994 or sacrifice child
Killed process 31204 (python) total-vm:7350860kB, anon-rss:4182920kB, file-rss:2356kB, shmem-rss:0kB