Right now our Jenkins agents generate a docker-compose.yml for each of our Rails projects and then run docker-compose up. The docker-compose.yml has a main \"web\" container
If you're willing to use docker-compose run to manually kick off your tests, adding the --rm flag, oddly enough, causes Compose to accurately reflect your command's exit status.
Here's my example:
$ docker-compose -v
docker-compose version 1.7.0, build 0d7bf73
$ (docker-compose run bash false) || echo 'Test failed!' # False negative.
$ (docker-compose run --rm bash false) || echo 'Test failed!' # True positive.
Test failed!
$ (docker-compose run --rm bash true) || echo 'Test failed!' # True negative.