Using docker-compose with CI - how to deal with exit codes and daemonized linked containers?

Question

Right now our Jenkins agents generate a docker-compose.yml for each of our Rails projects and then run docker-compose up. The docker-compose.yml has a main \"web\" container

Answer 1

Building on kojiro's answer:

docker-compose ps -q | xargs docker inspect -f '{{ .State.ExitCode }}' | grep -v '^0' | wc -l | tr -d ' '

1. get container IDs
2. get last runs exit code for each container ID
3. only status codes that does not start with '0'
4. count number of non-0 status codes
5. trim out white space

Returns how many non-0 exit codes were returned. Would be 0 if everything exited with code 0.

Answer 2

Since version 1.12.0, you can use the --exit-code-from option.

From documentation:

--exit-code-from SERVICE

Return the exit code of the selected service container. Implies --abort-on-container-exit.

Answer 3

docker-compose run is the simple way to get the exit statuses you desire. For example:

$ cat docker-compose.yml 
roit:
    image: busybox
    command: 'true'
naw:
    image: busybox
    command: 'false'
$ docker-compose run --rm roit; echo $?
Removing test_roit_run_1...
0
$ docker-compose run --rm naw; echo $?
Removing test_naw_run_1...
1

Alternatively, you do have the option to inspect the dead containers. You can use the -f flag to get just the exit status.

$ docker-compose up
Creating test_naw_1...
Creating test_roit_1...
Attaching to test_roit_1
test_roit_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)
$ docker-compose ps -q | xargs docker inspect -f '{{ .Name }} exited with status {{ .State.ExitCode }}'
/test_naw_1 exited with status 1
/test_roit_1 exited with status 0

As for the db container that never returns, if you use docker-compose up then you will need to sigkill that container; that's probably not what you want. Instead, you can use docker-compose up -d to run your containers daemonized, and manually kill the containers when your test is complete. docker-compose run should run linked containers for you, but I have heard chatter on SO about a bug preventing that from working as intended right now.

Answer 4

If you're willing to use docker-compose run to manually kick off your tests, adding the --rm flag, oddly enough, causes Compose to accurately reflect your command's exit status.

Here's my example:

$ docker-compose -v
docker-compose version 1.7.0, build 0d7bf73

$ (docker-compose run bash false) || echo 'Test failed!'  # False negative.

$ (docker-compose run --rm bash false) || echo 'Test failed!'  # True positive.
Test failed!

$ (docker-compose run --rm bash true) || echo 'Test failed!'  # True negative.

Answer 5

You can see exit status with:

echo $(docker-compose ps | grep "servicename" | awk '{print $4}')

Answer 6

--exit-code-from SERVICE and --abort-on-container-exit don't work in scenarios where you need to run all containers to completion, but fail if one of them exited early. An example might be if running 2 test suits in concurrently in different containers.

With @spenthil's suggestion, you can wrap docker-compose in a script that will fail if any containers do.

#!/bin/bash
set -e

# Wrap docker-compose and return a non-zero exit code if any containers failed.

docker-compose "$@"

exit $(docker-compose -f docker-compose.ci.build.yml ps -q | tr -d '[:space:]' |
  xargs docker inspect -f '{{ .State.ExitCode }}' | grep -v 0 | wc -l | tr -d '[:space:]')

Then on your CI server simply change docker-compose up to ./docker-compose.sh up.