Setup nginx not to crash if host in upstream is not found

Question

We have several rails apps under common domain in Docker, and we use nginx to direct requests to specific apps.

our_dev_server.com/foo # proxies to foo app
our_dev_server.com/bar # proxies to bar

Config looks like this:

upstream foo {
  server foo:3000;
}

upstream bar {
  server bar:3000;
}

# and about 10 more...

server {
  listen *:80 default_server;

  server_name our_dev_server.com;

  location /foo {
      # this is specific to asset management in rails dev
      rewrite ^/foo/assets(/.*)$ /assets/$1 break;
      rewrite ^/foo(/.*)$ /foo/$1 break;
      proxy_pass http://foo;
  }

  location /bar {
      rewrite ^/bar/assets(/.*)$ /assets/$1 break;
      rewrite ^/bar(/.*)$ /bar/$1 break;
      proxy_pass http://bar;
  }

  # and about 10 more...
}

If one of these apps is not started then nginx fails and stops:

host not found in upstream "bar:3000" in /etc/nginx/conf.d/nginx.conf:6

We don't need them all to be up but nginx fails otherwise. How to make nginx ignore failed upstreams?

Answer 1

1. If you can use a static IP then just use that, it'll startup and just return 503's if it doesn't respond.

2. Use the resolver directive to point to something that can resolve the host, regardless if it's currently up or not.

3. Resolve it at the location level, if you can't do the above (this will allow Nginx to start/run):

   location /foo {
     resolver 127.0.0.1 valid=30s;
     # or some other DNS (you company/internal DNS server)
     #resolver 8.8.8.8 valid=30s;
     set $upstream_foo foo;
     proxy_pass http://$upstream_foo:80;
   }
   
   location /bar {
     resolver 127.0.0.1 valid=30s;
     # or some other DNS (you company/internal DNS server)
     #resolver 8.8.8.8 valid=30s;
     set $upstream_bar foo;
     proxy_pass http://$upstream_bar:80;
   }
   

Answer 2

The main advantage of using upstream is to define a group of servers than can listen on different ports and configure load-balancing and failover between them.

In your case you are only defining 1 primary server per upstream so it must to be up.

Instead, use variables for your proxy_pass(es) and remember to handle the possible errors (404s, 503s) that you might get when a target server is down.

Answer 3

For me, option 3 of the answer from @Justin/@duskwuff solved the problem, but I had to change the resolver IP to 127.0.0.11 (Docker's DNS server):

location /foo {
  resolver 127.0.0.11 valid=30s;
  set $upstream_foo foo;
  proxy_pass http://$upstream_foo:80;
}

location /bar {
  resolver 127.0.0.11 valid=30s;
  set $upstream_bar foo;
  proxy_pass http://$upstream_bar:80;
}

But as @Justin/@duskwuff mentioned, you could use any other external DNS server.

Answer 4

You can do not use --link option, instead you can use port mapping and bind nginx to host address.

Example: Run your first docker container with -p 180:80 option, second container with -p 280:80 option.

Run nginx and set these addresses for proxy:

proxy_pass http://192.168.1.20:180/; # first container
proxy_pass http://192.168.1.20:280/; # second container