Change default route in docker container

Question

I have a docker container that is connected to two networks, the default bridge and a custom bridge. Via the default, it is linked to another container only in the default n

Answer 1

If I understand the question, the problem is : when restarting a container connected to multiple bridges, how to prefer a bridge to use for default route ?

I searched available options and made some tests, I did not found any docker command line option to specify a default route or to prefer a bridge as default when the container is connected to multiple bridges. When I restart a container connected to the default bridge (bridge) and a custom bridge (your homenet), the default route is automatically set to use the default bridge (gateway 172.17.0.1). This corresponds to the behavior you describe.

Solution 1: Specify a start script in the run command that is in charge to change the default route and start the service(s) you container has to run:

docker run \
  --cap-add NET_ADMIN \ # to allow changing net settings inside the container 
  --name container1 \
  --restart always \ # restart policy
  your_image \
  /path/to/your_start_script.sh

The your_start_script.sh:

ip route del default 
ip route add default via 192.168.130.3

# here goes instructions/services your container is supposed to run

This script has to be available inside the container, it can be on a shared folder (-v option) or loaded at image building with a Dockerfile.

Note: before connecting the container to your custom bridge (docker network connect homenet container1), your_start_script.sh will crash because the default route does not correspond to any available network.

I tested to log the output of ip route inside container1 run with --restart always, after connecting it to the custom bridge it has the wanted default route.

Solution 2: Set container default route from host on container start events

docker events --filter "container=container1" |\
  awk '/container start/ { system("/path/to/route_setting.sh") }'

Where route_setting.sh contains your instructions for changing the container's default route:

pid=$(sudo docker inspect -f '{{.State.Pid}}' container1)
sudo mkdir -p /var/run/netns
sudo ln -s /proc/$pid/ns/net /var/run/netns/$pid
sudo ip netns exec $pid ip route del default 
sudo ip netns exec $pid ip route add default via 192.168.130.3

This solution avoids giving special permissions to the container and transfers the route changing responsibility to the host.

Answer 2

@Silicium14

Thanks a lot for your 2nd solution. Took me quite long to find a way to set routes upon container start. I changed your lines a bit according to my needs as I need to provide a container name from docker events to the script

First I start the listener for my events.

docker events --filter 'container=box1' --filter 'container=box2' --filter 'event=start' --filter 'event=stop' --format '{{.Actor.Attributes.name}}'|awk '{ system("/work/route_setting.sh " $1) }'

I use more filters as I need the events for two containers of type start or stop Using --format one can control the output very nicely. So only the container name is piped to awk. Which then fires my routing script with the correct containername.

#!/bin/bash

# exit if no container name provided as $1
[ "x$1" = 'x' ] && exit 1
# holds pid of the docker container
pid=''
# read the pid for container
pid=$(docker inspect -f '{{.State.Pid}}' "${1}" 2>/dev/null)
# if for whatevery reason we get pid 0 avoid setting routes
[ "x$pid" = 'x0' ] && pid=''
if [ "x$pid" != 'x' ] ; then
  # let the routing happen 
  mkdir -p /var/run/netns
  ln -s /proc/$pid/ns/net /var/run/netns/$pid
  ip netns exec $pid ip route add 10.0.0.0/8 via 10.66.101.1
  ip netns exec $pid ip route add 192.168.0.0/16 via 10.66.101.1
fi
# clean up broken symlinks which occur when a container is stopped
# verify that your find supports -xtype l
find /var/run/netns -xtype l -exec rm -f '{}' \;

Answer 3

You can enter into the namespace of a container with the nsenter command, and then execute a command in it with. Around so:

nsenter -n -t $(docker inspect --format {{.State.Pid}} $dockername) ip route add something.
nsenter -n -t $(docker inspect --format {{.State.Pid}} $dockername) ip route del something.